Server side email attachment filtering

Lets say you have a ticketing system at work.

A ticketing system that does not like receiving large attachments

Where it will either crash on receiving a message that is too large or fail to create a ticket upon receiving said message as well as failing to notify the sender that something went wrong.

Obviously, the best solution would be to get a better ticketing system and avoid the whole thing...

Unfortunately,  You often end up finding out long after the product had been purchased and you are the poor soul left having to "make it work" anyway....

As you will soon see, it is fixable but will require a good amount of duct tape.

Most ticketing systems are setup like this

support@company.com  -->; ticket_system@ticketing.com

If you are lucky, the email server you have at your company is awesome and supports server side filtering natively. If that is the case, then

The other solution (besides buying a different solution) is to remove the attachments before they make it to the ticketing system.



Lets assume, that since you ticketing system sucks, then your companies email server probably also leaves a bit to be desired and there isn't a way to filter out the email attachments.


So, how do you make it work?
Add yet another server into the mix.

I know, hardly ideal for various reasons but I'm sure at this point you are desperate to get it to work.
Then the flow diagram changes to

support@company.com --->; filter@filterserver.com -->;; ticket_system@ticketing.com

You can filter it out using the sieve/pigeonhole language.

I spent sever hours looking, and I wasn't able to find any examples. Heck, I don't think the developers had ever intended sieve to be used in this fashion as what I needed wasn't really documented, but after trial and error I eventually had sucess.


Most systems do not support it (especially hosted email servers) but there is still a few.
Thankfully, favorite email company fastmail  supports sieves.

To jump to the case, here is the bits of code and what they do.

===============
require ["fileinto","envelope",  "reject", "notify"];

#If the email is addressed to support@company.com or emea_support@company.com then keep going, otherwise ignore this message (useful to ignore spam addressed to invalid addresses).

if header :contains ["to", "cc","bcc","resent-to"] ["support@company.com", "emea_support@company.com"] {
#As you can see, there can me multiple values inside brackets and can filter based on any email field


# When the email size is greater than 300k, remove all attachments while  appending (Attachment Removed) to the subject line.

#If it is still too big, truncate the body and bring message down to 100k
#You can also use M for megabytes.

if size :over 300K {
      notify :method "mailto" :options ["ticket_system@ticketing.com","From","Orig","Length", "300k"] :message "$from$ /  $subject$(Attachment Removed) / $text[100k]$";
discard;
#Need discard here, otherwise message will remain in the INBOX on the filter account, where it would eventually exceed your quota and you will be woken up at 3am with a nasty phone-call because the email suddenly stopped flowing

# Also Note discard is immediate permanent deletion, it will not get put into your trash folder. 
#If you want to do that, use fileinto "INBOX.Trash" (or equivalent folder name)
stop; #Stop possessing this message because we don't want to redirect the email twice with the lines below
}

# If it's smaller then that, allow the message to go through like normal without any truncation
redirect "ticket_system@ticketing.com";
stop;
==========
#If you just want to just generate a bounced  message, then do something like this

require ["fileinto","envelope",  "reject", "notify"];

if header :contains ["to", "cc","bcc","resent-to"] ["support@company.com"] {
if size :over 300K {
reject "This message is too big for our system (over 300k).
Please shrink the size of your message and try again";
discard;
stop;
}
redirect "ticket_system@ticketing.com";
}

Question:I have full bars on my wifi signal, why doesn't it still work? I was at a meeting with a mix of PCs and Macs but only the Macs work.

Simple Answer: It's complicated. Many different pieces of equipment, software and environmental conditions work together for WIFI to work. If any one of these are done poorly or get screwed up than you can get very erratic and inconsistent problems that become difficult to track down.

More sophisticated answer (while trying hard to keep it non-technical).

There is a of the common misconceptions on those signal "bars" on most computers wifi monitors. Most of them only list the raw strength of the signal, without taking the noise of the environment into consideration. Most cell phone bars on the hand take this into consideration and do calculations to come up with more accurate numbers (although certainly not perfect either).

One way to explain it is imagine that you are at a bar, with a band playing loudly. You try to talk to your friend who is next to you and they have to talk louder (sometimes even shout) in order for you to hear what they are saying. You can physically "hear" what the person is saying but because there is so much noise in the background (band, other people,etc) that you can not understand any of it. The wifi bars are based on the first measurement, not the second. It is also common where if the noise is loud enough, the sound will bounce off the walls and come back at you as an echo and wifi can have similar problems as well.

One tactic is where you start talking louder to talk with your friend, but then the people at the next table start talking louder to compensate for you, so you then talk even louder,etc. Wifi has adaptation techniques like this, which work adequately in simple situations but can break down in complex one's where everybody can end up shouting and nobody can hear anybody else. One option is to walk down the hall to where it is quieter and then have your conversation there. The same tactic also works with WIFI but it is not as easy/intuitive to do compared to talking in person as it is difficult to get to that information.

It is very possible that the office environment you were in had many sources of interference which could have made it very difficult for the computers to connect. Bluetooth, microwave ovens, wireless security cameras, certain Florissant lights and many more devices operate in the 2.4ghz band which can significantly increase the RF background noise.

These patterns also change over time increasing the complexity of the issue, so for example you had your first meeting during lunchtime where somebody was warming up their leftovers in the unintentionally microwave knocking everybody off but the second meeting was in the afternoon when the microwave wasn't in use and then everything worked fine.

Now back to your question - Why did the mac's work but no the PCs? There is many factors involved and it only takes for one piece to break for the whole system to not work. Certainly there is many software and implementation bugs that exist in the network protocol side of things (DNS/DHCP/Buggy NAT functionality) which can be another article by itself. It would be very difficult to know for sure what actually caused the problems, especially after the fact. I would also like to add the possibility of these RF environmental factors as well.

In my experience, Mac's tend to have excellent wifi implementations (especially non-mental one's). I have worked on many Windows PCs and the quality of the implementations can vary wildly, with many of the cheeper models having very poor performance and can result in erratic behavior. There are excellent wifi implementation in some PC models (certain DELL and most Lenovo models, off the top of my head) but they are not universally applicable.

I have an ASUS netbook that will function perfectly while in the same room as my AP, but once I move into my bedroom there is 100% packet loss even though it still claims to be associated with the AP and still showing full wifi bars. If I use my macbook, it will continue to work for the most part but will regularly get 25% or so packet loss and randomly disconnect occasionally (In Windows xp, the behavior is identical). Both show full bars, but because of the RF noise of the environment (apartment) and material of the walls the signals get distorted significantly. I have a friend who has a 17" HP laptop and gets maybe 45% packet loss in that room. It still works technically, but runs so slowly at that point it isn't really worth the trouble.

If the RF background noise was loud enough, * theoretically* it could prevent proper connections across the board. From my understanding, the process of associating (or initially connecting) to an access point is done in a special mode that runs at a significantly slower speed and therefore more resilient to interference. Once the stations have synchronized, then they transition to the normal full speed mode. If the signal is marginal in practice (based on RF noise not signal strength) than having a quality implementation could be /just enough/ to make the difference.

Believe it or not, placement in the room could also make a difference. In my room, if I move my macbook 2-3 inches in one direction it goes straight to 100% packet loss while still claiming to be on a strong signal. If I move it back, things return to normal. 2.4ghz is small enough that the signals regularly bounce off walls and other flat objects, sometimes causing those echo's to cancel out the original signal.

As you can see, it is indeed complicated. Sometimes there is things you can do about it and other times all you can do is switch to physical wires. Even under ideal conditions wireless technologies can never ever match the reliability and consistency of wired connections because it is the nature of the beast. However, you can try really hard and get it to "good enough" that you don't have to worry about it all the time and still be able to get your stuff done.

==========================================================
Definitions: (For those who need a little help understanding the jargon)

WIFI = WIrelessFIdelity, which is shorthand for the alphabet soup that is 802.11a/802.11b/802.11g/802.11n, which is a series of technical standards so that everybody can use the same "language" and work together. It does work pretty well for the most part, but that is because it assumes everyone will follow the rules (which is true for the most part, but not always).

RF=Radio Frequency

2.4ghz (i.e. 2.4GigaHertz or 2400Mhz) is the frequency that wifi normally runs at, along with many other devices. Like your car radio, each channel has it's own frequency. When they say "94.5 the rock", that number 94.5 is in reference to 94.5Mhz or MegaHertz. Depending on the frequency, the signal will behave in different ways physically, which depending on your particular situation can be an advantage or disadvantage. Simular to how AM stations seem to go further at night but are almost always have static. There is also another WIF frequency in the 5ghz band that has a lot more leg room to allow networks to operate without having to fight with each other, but it requires new/different equipment to be compatible.

For more, see http://electronics.howstuffworks.com/radio9.htm

AP =Access Point. It is a radio transmitter that is connected to your physical network and allows you to talk to it wirelessly. For most people, this is equivalent to your "router" at home but not necessarily with any of the smarts attached.

DNS = Domain Name Service. It is how your computer translates www.amazon.com into numbers that allows the computer to actually connect to the server on the other end by retrieving an IP Address.

IP Address = A series of numbers that is assigned to each computer on the internet and is something like a "phone number" you use to "call" the computer on the other side.

DHCP = Dynamic Host Configuration Protocol. When your computer first connects to a network, it probably doesn't know who to talk to in order to get online. So, it first shouts out "How do I get online?" and a server responds by giving you information how to do so (namely an ip address, gateway and dns server). It has a little table where it keeps track of who has which numbers to prevent problems.

Packet = The internet operates by breaking down everything into little pieces or packets, that are then sent out across the internet. Each one gets a number so that they are tracked and get put back together in order once they get there.
Packet Loss = I am sure you have had a time where you had a phone call that was really fuzzy/staticy and you had a really hard time understanding the other person and had to keep asking "can you say that again?". Simular things can happen with computers. This happens all the time and systems automatically compensate by asking the computer on the other end to send it again. A small amount (1-2%) is "normal" and expected, but each time that happens extra time is wasted asking for it again and piecing things back together. After a point, you spend more time dealing with mistakes than you do sending new messages and things then get really slow. Most often this is the result of noise on the line or the radio. This all happens automatically, so it is easy to get frustrated when we don't know that this is happening.

I would say most consumer ISP's provide pretty bad DNS service, and you can clearly tell that they are running slow when you spent a lot of time waiting for the browser to finish "resolving name www.xyz.com".

I am constantly surprised with how big of a difference it makes when I am working on other people's machines.

Here is their page with instructions on how to change yours.
http://code.google.com/speed/public-dns/docs/using.html

One alternative is opendns (if you don't like google).
Their service is good and even has a few features that google doesn't provide, but you are required to make an account to turn off their more annoying "features" that are on by default.

However, their help page does have extensive instructions that includes *pictures* of what needs to be done, which makes it really easy to figure out where you need to go

https://store.opendns.com/setup/computer/

You can use their instructions and replace their numbers with google's if you like - the steps are exactly the same.

Both have their own pro and con's (especially if you look at them politically) but google's servers are the least hassle to setup.

Computer Security

The bottom line is - the internet is DANGEROUS out there. There are tens of thousands of viruses out there, with many more people trying to get into your computer. Ten's of thousands of people every year get their identity stolen THROUGH the computer, because they got a trojan that then recorded their credit card #'s when they were ordering something online.

If you are running windows, you become VERY vulnerable if you
1) Don't keep up with ALL the updates. This includes, windows, flash, java and acrobat reader [there has been recent attacks that automatically infect your computer if you are using a older version, even if you have ALL the windows updates]. Yes- there is the risk of the updates failing and messing up your computer (regular system maintenance will /prevent/ that most of the time) but you just have to deal with it if you want your system to be half-way secure.

These system updates are the "first line" of defense. They prevent most (but not all) of the attacks that automatically infect you by visiting a website (without clicking on anything) or opening a email (without opening the attachment). The reason they can do that is because there is a hole in the system itself, and you need to constantly patch those holes ("patches") when they are discovered.

The reason you are getting the "windows advantage" thing is because your system has been updating (probably automatically) and MS sneaks that in there. Eventually, you are FORCED to "prove" that your copy of windows is legit before you can get more updates.

2) Current Anti-Virus with a VALID subscription
I have seen so many machines that had anti-virus installed (when they bought the machine) but the subscription had run out and the software deactivated.
Go and CHECK to make sure it is actually valid. If you have not PAID MONEY for a new subscription (yearly), then most likely it is expired.

If you don't want to spend the money, then get the FREE version of AVG anti-virus (http://free.avg.com).

You will have to manually upgrade to the new free version they release every year or so [to keep you on your toes], but that isn't that big of a deal.

This will protect against traditional viruses that you get by opening attachments and downloading files off websites.

Also, make sure to remove the expired anti-virus first, as the programs can conflict.
If you have problems, you can download and run these specialized remover tools which will manually clean out all the files hiding on your system. I normally do it routinely as incompletely removed A/V software can prevent people from getting online, among other issues.
http://www.technibble.com/repair-tool-of-the-week-antivirus-removal-tools/

3) Spyware Software
This is the part that gets more complicated, because there is so MANY spyware programs out there.
Recently, there has been a trend of spyware pretending to be "real" anti-virus, and giving you all these fake warnings about your computer being infected. After clicking on it (and "installing" the program), it then extorts you for money before it will allow you to REMOVE the software [and/or make the messages go away] (and in the mean time, it installs trojans and all kinds of nasty stuff to track what your doing).

Most people notice when they are infected, when their computer starts running REALLY slow (because there is so many spy processes running, that it takes more computation time to do /that/ then it is to run your real programs). Another is when you get random pop ups for advertisements when you are not even online.

There is NOT one single program that can take care of all of them. Every few months, the programs that were effective fall back and new programs come into the scene that work better. I will just list the stuff that is FREE, as anyone can use it.

A) Spyware Doctor STARTER edition (Not "FREE" trial edition) - as part of the google pack (you can install only that and and uncheck everything else
http://pack.google.com
A good solid program that runs scans every once in a while by itself.

B) Malware bytes - The free version only supports MANUAL scanning (so you should run it once a month or so).
It is very effective and fast
http://www.filehippo.com/download_malwarebytes_anti_malware/

C) Super Anti-Spyware
The name makes it sound fake, but (so far) a lot of people have been having good success with it (including myself).
There is other programs that I would run myself after this, but they are more complicated to use and setup then this
http://www.filehippo.com/download_superantispyware/


4) Backup!
ALL Computers die all the time - especially hard drives [yes, even mac's].
Make sure to back up your pictures and documents regularly (and whatever else you can't replace).

5) Be Suspicious!
If you are browsing a website, and a pop up randomly comes up and says you are infected (citing a name of a program you have NEVER seen before).
Be very suspicious. You would not eat pills from some random stranger you met on the street - it could potentially be poisoned.
Go and VERIFY who it is, and make sure they are someone safe to get stuff from before you go and download a program.
You go to a doctors office or a pharmacy to get medicine.

Don't EVER give out your password online. If a tech support person needs to get into your account, they will have a way to bypass the password check and get in directly.

-------
If you DON'T like having to deal with all this, then get something else. Just because everyone else has a pinto doesn't mean it is the best car in the world. Get a mac or run linux (both of which have more stable systems and less viruses - for now, anyway). Windows CAN be safe and stable IF you are VERY careful with it and religiously stay up to date. It can be done, but it takes EFFORT to do so. Sometimes you need to run specialized software that is windows only (go and look - you can be surprised what has been ported over) and you just have to deal with it if you want to keep using that system.

A lot of people complain about mac's cost more, but if you INCLUDE the cost of time, energy and (occasionally) money to keep a windows machine secure (regularly getting it checked up by a professional), it can easily cost as much or more then a mac. I am NOT saying one is "better" then the other - I am saying it is a CHOICE, where there is plusses and negatives for BOTH and it is up to you to decide.

WOW! This got long really fast (hope this isn't too overwhelming for some of you), but there is a lot of stuff you have to be aware of.
Hope this will help some of you.

All of the above will protect you against 80% or so of the stuff that is out there.
It doesn't stop EVERYTHING, but it is a good start and some times the only defense you have is your head.

The other day, my brother asked me about setting up a webpage for his local group, so I came up with this quick evaluation of a couple of different options, and am throwing them up here for prosperity.
--------
There is a couple of different options that are available out here and they generally follow the line of
The simpler it is, the cheaper it is. Which has the plus side of making it easier to manage a website yourself and properly the next person after you will be able to use it.

Common options

Free
1) Google Apps/Pages
+ Free
+Really Easy to use,
+Still looks pretty damn good,
+supports multiple users (with some tweaking) - and even real time editing!
-How much you can customize it
2) Blog
+ Generally free,
+Easy to use,
+supports multiple users out of the box, for adding comments and new posts
+RSS
- VERY simple pages,
-Hard to make "static" plain pages that stay the same [for like standing schedules, routine documents,etc]

2)Paid Hosting somewhere (nfshost.com, pair.net,etc)
+Maximum amount of customization,
+can have a database online that can pull and manipulate data exactly as you like,
+can do all kinds of fancy and shiny things
+If done correctly, can look REALLY professional
-A lot trickier to setup
-If the one "tech person" who setup leaves the group, you're pretty much screwed most of the time
-or can look REALLY cheesy if done poorly

3) There is also "free web hosting" out there, that sticks advertising's onto your page and have all kinds of limitations that decrease of a lot of the benefits you would otherwise get.
Hosting from a company like nfshost.com is really cheap for small sites (often less then 50 cents a month) but still allows the same level of customization that you can get from one of the bigger companies.

Without you telling me explicitly what your needs are, I am going to suggest you give google apps/pages a shot.

If you already have a free google/gmail account, then I would say go to (http://pages.google.com ) and play around with the templates and see how the editing works, and how you like it.

------
If you DO like it, then your group can purchase a domain name (namecheap.com, godaddy.com, among MANY others) for about $10/year and then sign up for google apps and link them together (copy and paste a few numbers).

It gives you more features like..
Such as - Web Hosting, Email Hosting, Group mailing list, Shared Calendar's, Shared Documents and more.

Go checkout this description page [those videos do a good job explaining the benefits]
http://www.google.com/apps/intl/en/business/collaboration.html
http://apps.google.com
http://www.youtube.com/watch?v=wY2bpr1TAA4

and then sign up for the free version (which has gotten really tricky to find, recently)
http://www.google.com/apps/intl/en/group/index.html

You can do basically the same thing with a bunch of the free gmail accounts and sharing each new document/page you make with each other manually.

The main advantage with the app version (with a domain), is that you guys can have one unified web presence for your group, with one shared folder between all your members to keep track of everything.
You can send and receive email's using that same name, so it is clear which group you are representing when you are interacting with outside groups.

So for Example, lets say you could get example.com

people would go to http://example.com to see your webpage
if they wanted to send email, it would go to whoever@example.com

In the end, not that big of a deal, but it is surprisingly easy to setup and can make you guys look a lot more organized (if that is what you are going for).

How can you go to costco, spend over $400 in groceries and *still* not have anything to eat for dinner?

On my way home from school, I heard on NPR a piece about the lenovo (I.E. IBM Thinkpad) being produced in china scandal.

The bottom line is this.

1) All personal computers today, contain at least *some* chips that are produced overseas. Most companies (Apple, HP) are entirely built overseas. US designers, foreign manufacturers. IMHO, any personal computer has about the same chance of having a "bug" installed because the actual construction is done outside the country. Who owns the company on the paper has very little to do with any of this.

For the few that are "assembled" in the US (dell), they still buy their chips and components from overseas. So it is entirely possible that china snuck in the "bug" before it even arrived to dell.

It is entirely possible for someone to "drop in" a "bug" while they are putting the parts together. Not to mention that most of the common chipset designs have become standardize and therefore, some were designed in china as well.

2) Even if there was a company that built and manufactured all of its own components and chips in the USA, It is entirely possible for a Chinese agent to sneak in and put bug into that operation. You could even go on and say the same thing if the government went out and made their own machines in house for the same reasons.

Is it possible technically for there to be bugs? Yes, just as it is possible for there to be bugs in our clothes, food and cars. Just because they can, doesn't mean they are going to. The major problem facing the US intelligence agencies today is that they have *too much* information. They don't (or can't afford) enough computers and workers to sift through the countless hours of material they have. Once they hear of a problem, they can go back and search but they can't just passively look for "problems" and expect to have a effective return on the thousands of man hours necessary.

So this is entirely a publicity stunt by US politicians, who don't know about the issue, trying to look like they are "protecting" America. They are just being too paranoid on what little they know. To be honest, if it was really that big of a security issue one of the Three Letter Agencies would have already stopped them.

On to other news, it looks like shinyfeet.com is shutting down.

Streamload appears to be doing pretty well. They have recently upgraded all of the existing space plans (giving more space) and now ftp is now a official feature. They are not quite-yet done with their new version mediamax, but will be "soon". Interestingly, media max provides limited storage space while current customers are unlimited. They have repeatedly insisted that all existing customers will keep paying the same price and will keep their unlimited storage space.

The new version will only count the file storage quota to files that are unique to you. I.E. If you and a friend have the same file, then the file storage is free. I wonder if its free for both or for only one? If the first still "pays", then what happens when they delete? The second person in line then pays?

I am quite happy with the service pricing as it is now, and I don't know why they are (essentially) punishing new users (if they follow their plan). Maybe they have something up their sleeves (its 250GB storage per month? A button for long-term Archival storage?). The only way to be sure is to sign up to a paying account now (to hang onto the existing plan) and see what happens.

A LOOONG story short, I was looking around in the (US) miltitary medical requirements document (http://www.usapa.army.mil/pdffiles/r40_501.pdf) I.E. what condition can get you kicked out.

Most of it is pretty obvious (must have all your limbs, can not be blind, etc) there is a surpringly quite a few unusual things in there. Some I might say are trivial. I guess its to cover their buts legally if/when a really unsually case comes along.

You can be kicked out for
frostbyte
motion sickness
having too much acne!
warts
being a organ transplant recepiant!
having a history of sleep walking
Migranes

thought you would like to know ^_^.

Well, I've been trying out [shinyfeet.com] and it hasn't been working so well so far. They supposed to provide "Unlimited" email and file storage space. I found it off a DSLreports posting, where someone recommended them. However, I created two accounts and both can login into the webmail-interface and send email, but I can't receive any email or store any files. It seems that the "home directory" wasn't created. When I send a email to myself, it says my username doesn't exist.

Another alternative (FYI) is [streamload.com]. Used to provide unlimited space for free but now its only 10GB. Their gimmick is that you can only download 100MB/month unless you pay a mothly fee for more access (which also provides unlimited storage). However (at the moment) you can store or send files, you can't receive attachments by email (yet). They are currently updating their system so they might add that feature. I've created a little shell-script that serves as my backup that sends my data (in split up archives) to a email address [mainly because so it can easily be automated]

Blockbusters in-store pricing

At least, in my "town" of Saratoga, CA
$25 for unlimited DVD Movie Rentals (2 at a time)
~$30 for unlimited game rentals (1 at a time)