Keepin' buzy

Been pretty productive (read: teh buzy) the last 6-7 months, doing some (quite heavy) service- and maintenance-work for a local IT-startup.

Been responsible for the x86_64 Linux-rack(s) based infrastructure and backend driving the various product-frameworks, the domain- and network-security, plus client host-security throughout the organization. Quite a lot of networking and some aggregated WiFi-APs etc.

The Linux-rack servers have also been host-hardened and properly secured (SSH/TLS/SSL) for the threats facing online services in 2015.

Without going into too much detail, it all revolves around embedded GPS-tracking, web-apps for viewing and controlling said- tracking system, and the accompanying maintenance / service / troubleshooting involved in said systems and their server-backends.

The servers run both the MariaDB and the PostgreSQL database-systems for serving application- / geo- / PostGIS-data in various parts of the application-flow. I became responsible for servicing / doing maintenance of and fascilitating import / export of SQL-data for backup(s), re-location, and the like. The PostgreSQL database is even served from an iSCSI pool through a dedicated jumbo-frame ethernet connection to a NAS-rack for extra speed when handling huge datasets.

I've also been playing around with the web-framework, doing some development for myself to learn the technology, and also fixing various components in the active applications.

Fail2Ban

Fail2Ban works by scanning through log files and reacting to offending actions such as repeated failed login attempts, by using iptables to generate blocking-rules for any defined (listening) protocols / services, aimed at specific offending IP-addresses.

I used to utilize DenyHosts, but as the project was discontinued I had to adapt. And so I also had to retract all my recommendations of DenyHosts and update them all to endorse Fail2Ban instead.

If using Ubuntu or Linux Mint, setting up and using Fail2Ban is easy.

It comes pre-configured (on Ubuntu) to detect malicious SSH-activity with basic notification action,

Firstly, you just have to apt-get it:

sudo apt-get install fail2ban

Then, you just copy over the standard (Ubuntu) "skeleton"-config:

sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

 Then stop and (re-)start Fail2Ban to load and run the config:

sudo service fail2ban stop && \
sudo service fail2ban start

It can be customized to send e-mail alerts to designated addresses, and various other "actions_".

Protocols and / or services are easily added to the config-file if they aren't already present. Fail2Bans config-file uses an easy syntax (layout-format) for somebody with basic networking- and logging-knowledge.

I tend to also enable the "ssh-ddos" detection, since these days we're seeing more and more distributed attacks. There are more SSH-specific detection's, but they're not necessary.

Best desktop gigabit switch

The Netgear GS108 has been a rock-solid, 24/7, always-on, seriously dependable Internet switch-point for office / desktop use.

I have been using one as an internal gigabit switch-point between gigabit LANs and a FTTH fiber-gateway for 5+ years now. Not one outage!

It is, however, a dumb switch. Meaning: it is not a managed smart-switch, but rather more of a "stupid" repeater-block. But an extraordinary one at that :)

gethostbyname()

The last few years (2010-2015), I have been using a command line log-compiler tool I made, called "sshlog", to monitor incoming SSH-logins, both accepted and failed.

It is basically a crude administration-tool (server-side) for remote use on mobile devices and tablets. In combination with "DenyHosts" (python-program for blocking IPs that carry out repetitive and malicious ssh-bruteforce attempts), it proved to be a very solid security-framework.

That is, until January this year.

Suddenly sshlog reported that there weren't made any malicious ssh-connections the last month, and, DenyHosts did not add any new IPs since december last year. Weird...

While log-surfing and manually executing my sshlog-algorithm on my system-logs, I noticed there HAD been malicious ssh-bruteforcing (like usual)... but none of the offending addresses had been added to the blacklist because 'auth.log' was full of warnings about unresolvable hostnames in '/etc/hosts.deny'.

After checking out /etc/hosts.deny, it became apparent that DenyHosts had been adding random canonical hostnames (which, in turn, requires DNS resolving of the IP-addresses) instead of legitimate IP-addresses.

I made my discovery thanks to this repeating error in /etc/hosts.deny:

warning: /etc/hosts.deny, line xxxx: can't verify hostname: getaddrinfo(<some-dynamic-ip>, AF_INET)

*SIGH*
Firstly, I had to stop DenyHosts, then I had to manually 'sed' all the IP-addresses that had been added as canonical hostnames from the DenyHosts config-files, then from /etc/hosts.deny and lastly start DenyHosts up again...

The whole ordeal was apparently caused by none other than... myself -_-

I had enabled DenyHosts to do hostname-lookups for blocked IP-addresses :P *Doh!* which in turn forced DenyHosts to add dynamic hostnames to /etc/hosts.deny, no matter if they were legitimate addresses or not.

Fact is; many of these malicious SSH auto-dialers use dynamic IP-addresses that won't resolve as legitimate addresses in DNS (no registered reverse-lookup, or similar), thus, they are added to '/etc/hosts.deny' with (invalid) canonical hostnames that don't correspond to legitimate MAC and IP-address pairs.

Open networking; leaf-spine...

I recently came across a norwegian tech-article regarding open networking vs proprietary networking.

More specifically, it talks about the Open Compute Project - for those of you who don't know, read this. More information can be found here.

The project discusses the use of open standards when building huge, energy-demanding and efficient data-centers. Basically, OCP talks about replacing the common standard networking architecture with something (kind of) new, to meet the ever evolving (and consequently; demanding) bandwidth and streaming of tomorrows data-centers. Mainly, replacing the tree-span network-design with the leaf-spine network-design.

A leaf-spine network design consists of the following:

• Top-of-rack (leaf) switches (leading to spine-switches).
• Spine-switches (between leaf- and core-layers in a leaf-spine network).
• ... and other hardware and software solutions for rack-use.

LEAF-SPINE NETWORK

In a so-called "leaf-spine" layer-2 network; the spanning-tree protocol is often replaced with either a version of Transparent Interconnection of Lots of Links ("TILL") or Shortest Path Bridging ("SPB").

In a leaf-spine layer-3 network; each link is a routed link. Open Shortest Path First ("OSPF") is often used as the routing-protocol to compute paths between leaf and spine switches.

The norwegian in me thinks the reporter has (way too) big expectations, especially if he's counting on norwegian authorities and heavy industry for this (quite HUGE) change to take place.

But on the other hand, my inner sysadmin really likes the possibility of administrating a data-center with fully open networking-protocols / -equipment and/or -hardware :)

---

My own little µ-datacenter is based on open protocols, open systems, and somewhat open hardware.
http://blog.pizslacker.org/2014/10/datacenter-so-far.html

Norwegian article:
http://www.cw.no/artikkel/kommentar/apent-ikke-proprietaert

µ-cloud, so far...

Sporting as a: VM-lab / Workstation / A/V-rig / Gaming-rig / VM-production :P

3 years - 24/7 operation (2011-2014), and still running like a well-greased steam-engine :P everything runs some form of Linux ;) open standards - open systems :D all the way.

I've never had OP processing-capabilities before, especially at my own complete personal disposal (20+ years of computing-experience), until now :P 3,5 years later (that's including half a year of planning before starting the build).

Never going out of CPU-time, practically never overflowing (swapping) the system memory. And rarely even getting into CPU-loops (unless I'm doing some risky low-level programming for lulz).

I am proud to say that my domain is fully OSS-operated ;) :D

---

Network-wise, I have separated internal traffic into separate VLANs, mainly for layer-2 separation of different protocols and various traffic-types, like: Internet-traffic, incoming web-server traffic, VPS' Internet-traffic, management sub-net and iSCSI sub-net.

iSCSI even runs on separate NICs (on both storage-box and server), through it's own switch-block and separated sub-net, to prevent interference with other packets running on high-traffic sub-nets (like Internet-access).

Outgoing traffic (and incoming of course :P) is pumped through a VPN / Firewall network-box that has a firewall-throughput equal the total speed of the FTTH Internet-uplink connection ;D

---

h3x4c0m-build: http://blog.pizslacker.org/2011/12/h3x4c0m.html

Digital Attack Map

A real-time overview of world-wide DDoS attacks :P

Digital Attack Map screencap, taken @ thursday Oct. 24 2013 - 2:48pm

Link:
http://www.digitalattackmap.com/#anim=1&color=0&country=ALL&time=16002&view=map

sshlog v1.6

Bash scripts for generating and viewing simplified SSH access-logs in a console.

My reason for creating sshlog (and consequently sshloglist and sshlogviewer) was to have an easily available command-line toolchain, capable of generating and viewing simplified SSH access-logs through a console (plain-text interface, usually SSH+Bash).

sshlog generates a logfile based on command-line options given, then pipes the results to sshlog-viewer. Or, it can simply pipe the results into a timestamped text-logfile, in a directory called "ssh-logs" in your home-directory.

https://code.google.com/p/sshlog/

sshloglist is used to generate a box-based, selectable list of the logs already present in the ssh-logs directory, when a log is selected, it pipes the selected list filename to sshlog-viewer.

https://code.google.com/p/sshlog/

sshlogviewer is a box-based log-viewer interface. It prints the content from a given sshlog (from a datastream piped directly by sshlog, or given as a filename at the command-prompt or by sshlog-list). It then lets you flip through the sshlog-pages with the Page Up and Page Down keys. Press 'q' to quit (using less, if dialog is not installed) or spacebar/enter to press the "OK" button (using sshlog-viewer + dialog).

https://code.google.com/p/sshlog/

NAS for iSCSI/media/files

1.8TB SATA3 in RAID-1 configuration.

I have now switched my ReadyNAS Duo v2 with a ReadyNAS Ultra 2 Plus, which is the business-edition of Netgear's 2-bay NAS boxes. Way better equipped for my needs.

Hardware:

• Intel Atom dual-core 1.5GHz (x86 instead of arm5 as in Duo v2)
• Dual Gigabit RJ-45 LAN ports
• 2 x USB 3.0 on back-panel (one over each gigabit LAN port)
• 1 x USB 3.0 on front-cover (for "backup"-button)

Accessories:

• 2 x Western Digital Caviar "Green" 2TB SATA3 hard-drives

The Ultra 2 Plus business-ready revision of the dual-bay NAS from Netgear offers more advanced configuration and customization options, and also complies with Linux networking protocols (NFS, SSH, SFTP, etc.), unlike the Duo v2 which only offers AFP and SMB file-sharing (Apple and Windows network file-sharing respectively) and basic configuration.

What really makes it stand out is its ability to share iSCSI targets. Basically, it can share portions of its RAID-array as virtual hard-drives for Virtual Machines run either by for example VMware or VirtualBox on a server, or via a bare-metal hypervisor (ex.VMware ESXi, my personal favorite ;).

Extra functions in the Ultra 2 Plus compared to Duo v2, are for example the possibility for scheduled RAID-scrubbing and/or volume checking, to keep your RAID volumes clean and fixed :)

Global broadband speeds

...I thought it was a joke when people said we (Norwegians) have world-class Internet connection speeds. As I'm personally utilizing a 30Mbps fiber-connection, I've grown used to "instant access", and the concept of "lag" is far off from my list of worries nowadays.

But it wasn't always like that. I also remember the times of modem-breath, ISDN and yes, even radio-transmission-Internet (this was offered to people in my area who didn't have access to DSLAM-enabled phone-centrals (more commonly known as ADSL)). So I do have experience doing the "click-and-wait-game".

But after researching statistics, documents, ISP websites and various other information sources online, I was quite surprised.

Most countries in the world have to settle for average speeds UNDER 4Mbps! 4Mbps was the national MINIMUM for an ISP to even be able to call their service "broadband" in Norway 4-5 years ago(!). As this link shows, Norway along with a select few other European countries is in the top-percentage of high-speed broadband-Internet availability(!).

Only beaten by a small percentage of American ISPs offering 1Gbps uplinks.

Ironically, various ISP-companies in Norway are now considering building 1Gbps backbone-rings for public access.

I really, REALLY like current events! ;D Nerdalicious!

Google >> Linux-kjernen

Google har vist seg å være en ekstraordinær forkjemper for OSS i det siste, og legger mer gull i potten. Nå har de til og med bidratt med kode i Linux-kjernen for å forbedre nettverks-gjennomstrømming på fler-kjerne maskiner("Dual-Core","Quad-Core") (fler-kjerne maskiner kan håndtere mye større datamengder, raskere enn tidligere).

Internett og datanettverk generelt hopper voldsomt på hastighetsskalaen også i disse tider og ethernet-industrien legger om til nye hastighetsstandarder for utstyret de produserer (henholdsvis 40 Gigabit og 100 Gigabit nettverk). Forbedringene i Linux-kjernen vil da bidra til at prosessorene klarer å holde tritt med de enorme datamengdene og hastighetene som vil komme i tiden fremover.

Bedre føre var ;)

Link:
http://www.idg.no/computerworld/article174055.ece

Skype åpner kildekoden sin

Skype satser nå på variasjon og innovasjon ved å utvikle en åpen kildekode versjon av IP-telefoniprogrammet "Skype". De har ikke kommet med spesifikke detaljer rundt prosjektet, men understreker at det er Linux-klienten (det grafiske brukergrensesnittet) som nå åpnes. Ifølge utsagn fra selskapet, virker det ikke som det er snakk om frislipp av hverken hele programkoden eller protokollen.

"Et åpent grensesnitt vil hjelpe oss å få bredere appell blant de mange Linux-distribusjonene, samt andre plattformer"

Skype for Linux har alltid ligget bak tilsvarende teknologi for Windows og Mac. Men dette er nå i alle fall et steg i riktig retning i forhold til hvordan de lå an tidligere.

Link:

http://www.digi.no/827540/skype-aapner-linux-klienten

Dilling

Kjøpte som sagt ny router for å utvide flaskehalsen noe inn mot DMZ fordi jeg fikk en del falsk trafikk som belastet brannveggen på den offentlige serveren. Etterpå virket alt helt ålreit i noen dager, helt til den plutselig ikke svarte på requests den 3. dagen.

Når jeg kom hjem viste det seg at routeren ikke kjørte fastlagring på MAC-adresser, eller noen form for IP-liste. IP-adressen til serveren skiftet hver 3. dag, og det var ikke mulig å reservere adresser via web-konfig, irriterende. Etter litt manual-skumlesing viste det seg at akkurat denne router-typen (wrt54g2) ikke hadde nok minne til å reservere noe som helst, eller lot seg flashe med hjemmekokt firmware (chip-programvare) uten en nokså intrikat forberedelses-prosess.

Løsningen ble da relativt enkel:

• sett statisk IP-adresse på serveren...
• ...kjør port-forward via router...
• ...og håp på det beste.

Ting skal gjøres enklere å bruke, men omgjøres likegodt til "dumbokser" i prosessen og ødelegger dem for oss som faktisk vet å konfigurere og implementere. Jaja, kan ikke få i både pose og sekk...

UPnP media

If you want a complete newbie how-to on making a spare linux box your mediaserver, this link may be helpful.

I, on the other hand, am utilizing Slackware Linux ("The Swiss army knife of Linux") to suit my serving needs. I won't touch Ubuntu with a ten-foot pole, mostly because they focus on bleeding edge technology, which doesn't necessarily mean "stable".

And, I use "FUPPES" (Free UPnP Entertainment Service) instead of mediatomb. In my opinion, FUPPES is better equipped to transcode untraditional coded video- and music-formats. And when I tried to use mediatomb, I had trouble recognizing the service on my PlayStation3, no matter how much I tweaked the config.

To put it quite simple, I compiled FUPPES with only traditional Slackware stock-packages and libraries (where mediatomb demanded I compile a plethora of non-standard Slack-packs), added the device-IPs to the whitelist of FUPPES and opened FUPPES-designated ports to my local LAN subnet.

Thus, limiting access to the service from anything other than my internal LAN hosts.

Things FUPPES does that mediatomb does not (or at least not on my Slackware-machines anyway):

• Converts untraditional coded video to standard-formats
• Plays DivX and XviD off-the-bat without hassles what-so-ever
• Plays almost ALL audio-formats with standard libs
• Displays every image-format I have on-disk

Linux servers

Well, it finally happened. My long-time friend "slamd", the experimental Linux server box, has died.

( The picture shows the pub-machine 'slackr' )

I got it from my brother about 4-5 years ago, when it had been obsolete for almost 2 years already. It was not a custom power-house, nor was it bleeding edge technology.., it was MINE. And it did what I had in mind for it, and it did it pretty damn well too I must add.

It was a scruffy AMD Athlon XP 2500+, running on a special legacy ATX-board made by 'ABit' --not the most stable hardware I've ever used--, with 2GB Corsair DDR RAM, a GeForce 6600 GT GFX card, some USB 2.0 extension cards and a Creative Audigy 2 ZX 7.1 digital surround soundcard.

I used the box as:

• a house-central secondary NAT router
• an UPnP multimedia server
• a network fileserver
• a central database server (for hosting web-content served by my public-machine)

Lately, the motherboard just wouldn't boot up properly. My suspicions fall on the rather dusty CPU-chip I had put in it, but for all I know, the motherboard has met it's end of days...

So, I had to disconnect my public-machine (a dusty, dirt-old Pentium III 800MHz box), throw out 'slamd', put in the public-machine as a temporary replacement and route all the public requests in to my private internal LAN where the pub-machine acts in it's own DMZ.

Then I had to re-compile my UPnP mediaserver software because 'slamd' had it's software repository stored on one of it's internal IDE harddrives, and I just couldn't be bothered about doing backup-jobs and the like... so, I went with 'K.I.S.S.' like I usually do ;P

OS X utsatt for botnet-orm

http://www.itavisen.no/811025/forste-botnet-for-os-x

Nå er det i hvert fall ikke trygt å bruke Mac lenger...

Mac / Apple har siden 70-tallet vært trendsetter på data-markedet, så det var ikke et spørsmål om "hvordan" eller "hvis", men "når". Visste det bare var et tidsspørsmål. Og det har jeg sagt lenge, så lenge det finnes en merkbar markedsandel med folk som bruker visse typer systemer, vil disse da bli mål for cyber-kriminelle (crackere).

Botnets har lenge vært et problem med Windows-maskiner. Mac-folket har lenge ment at noe tilsvarende ikke kan skje med deres maskiner. Dels fordi de er en minoritet få virusmakere gidder å bry seg med, og dels på grunn av at Apples operativsystem OS X er bygget på Unix - en plattform som av mange regnes som sikrere enn Microsofts.

En nedlastbar fiks fra Apple, kalt "iWorksServices Trojan Removal Tool", kan hentes ned herfra.

For å videre beskytte Mac- / Apple-maskiner fra slike trusler, bør en installere anti-virus og evt. anti-spionvare programmer (og her gjelder KISS-prinsippet, mer enn ETT anti-virus eller anti-spionvare program på samme maskin byr på mer problemer enn nytte).

Travelling companion

My EPC 900 on a food-tray, in one of NSB's new "Stadler Flirt" cross-country train models.

"GhostNet" linked to the "Waledac" botnet

Conficker.C-infected computers have shown activity recently, according to security analysts and software/network engineers, so it seems the threat is not over...yet.

It's main activities (identified activities, that is), are:

• downloading (malware from other botnets, mainly the spammer-botnet "Waledac", better known as the re-animated Storm DeadNet "Valentines e-mail spammer botnet")
• linking (assumably to other malicious botnets)
  
• communicating (assumably with it's creators).

It is also reported to flash rogue anti-viral software ads directed at users of these infected machines.

Darknet.co.uk had this article to explain (excerpt from article below):

“Fear is used, universally, as a means to control people,” said Sendio CTO Tal Golan. “Governments use it. Large businesses use it. So it should come as no surprise to anyone that ‘cyber-bad guys’ use it.”

At the moment, the rogue anti-virus software comes from sites located in the Ukraine (131-3.elaninet.com.78.26.179.107) although the worm is downloading it from other sites, according to Kaspersky Lab.

Hmm. No, not surprising at all if you ask me.

Norsk - 60 Teraflop Switch

Norsk svitsj gir 60 teraflops på Blindern - digi.no : Bedriftsteknologi

Svitsjen utgjør hjertet i superdatamaskinen «Ranger» ved Texas Advanced Computing Center. Ranger kom på fjerde plass på Top500-listen allerede ett år etter lanseringen av Constellation 3456.

Slik ser Constellation ut før den koples til.



Slik ser den ut tilkoblet.

The "Conficker.C"-hype

I've read like tenfolds of articles both international and national about the suspected D-day, April 1st 2009. It is rumored that over 10 million computers will release a tidal-wave of DDoS-attacks on a global scale...

However, if you secured your home-setup earlier/today (like I have), you SHOULD be safe, for the time being...

Time will show.

But... to give you an idea about HOW malicious this worm is, Microsoft has promised a $250.000 reward for anyone able to provide information about it's creator(s). So, it's a serious issue.
Or...is it? ;P (April 1st 2009?? C'mon...)

---

For mine norske lesere, her er en ganske detaljert artikkel om "problemet": http://www.dagbladet.no/2009/03/31/kultur/tekno/internett/virus/5549841/

For å utfylle litt mer ut ifra det engelske innlegget ovenfor:
Er du sikret mot Botnets, som f.eks. ved å bruke OpenDNS istedenfor en lokal navnetjener (router, gateway, e.l. som cacher navnespørringer lokalt, utgjør en sikkerhetsrisiko på IP addressering mot Internett), så er du rimelig sikret i tilfelle noe av dette finner sted. Lite trolig, siden det dreier seg om 1. april...