Darwin’s evolution

19 Feb 2009
by: By John Adams
A vigorous oasis growing fast through the joint economic drivers of tourism and mining, Darwin’s progress is mirrored by SKYCITY Darwin, a low rise development with its feet in the jade Arafura Sea. Even mid-week it’s easy to see why this thriving business needs serious surveillance capabilities.
EARLY adopters don’t always get the best of new technology. It’s usually smarter to hold off and make a measured selection when time has scrutinized shortcomings and polished out flaws. SKYCITY Darwin held out longer than most – just 12 months ago its surveillance system was all analog and staff juggled VHS tapes recording video streams from more than 300 cameras.

 
In hindsight the delay in adopting digital technology was fortuitous. Instead of wrestling with a major hybrid solution incorporating elderly matrix switchers, SKYCITY Darwin chose Dallmeier’s SeMSy Video Management system supplied by C.R.Kennedy and installed by ACE Cabling. The SeMSy solution is supported by Dallmeier’s DIS-2 MPEG-2/MPEG-4 Dual Streaming encoders (running on Linux), and a selection of quality off-the-shelf network hardware. It’s a solution that’s as elegant as it is functional, a design that truly keeps storage at the network edge.

Important to consider is that SKYCITY Darwin’s new digital system needed to be more than just capable. Darwin is a long way from major distributors and integrators in the South, and like all casinos, the solution’s design brief demands always-on surveillance buttressed by an underlying capacity to perform under virtually any circumstances.

Part of this system-toughening process is ensuring the casino’s inhouse technicians know how to install and adjust cameras and other components in the event of failure. While unusual, this training guarantees the casino’s operational viability has signification protection in the event of failure – that’s important because casinos in Australia can’t legally operate without surveillance. 



 “I couldn’t put a figure on the improvement the switch to digital has brought us. However from the point of view for staff productivity, it’s huge. Digital means more hours of the staff observing the monitors and protecting the assets of the casino”


 

Like most modern systems, the new SKYCITY Darwin solution incorporates analog CCTV cameras, which are carried via coaxial cable to the server room and then funneled into Dallmeier DIS-2 encoders. The moment video arrives at these encoders, resemblance to other networked surveillance solutions ends. The reason for this is that Dallmeier’s DIS-2 encoders sport a pair of onboard 250GB hard drives between the encoder and the network interface card.

Each DIS-2 supports a single input and 8 days of full frame MPEG2, 25ips video is stored in an onboard RAID-1 mirror, with selected incidents directed to an archive server for long-term RAID-5 storage. The result of this architecture is profound. There’s more than 180 Terabytes of hard drive storage at SKYCITY Darwin yet the surveillance system runs on a Fast Ethernet LAN.

Analog past

The former analog system which incorporated a 1000 switcher and oodles of attention seeking VCRs was installed in 1995 and according to SKYCITY Darwin’s security, surveillance and property services manager, Bill Page, it was the best  of CCTV systems in its day. By 2007, however, its capabilities were distinctly pedestrian and there was a real need to leverage the efficiencies and functionalities of a digital solution. 

Page, who began his career in the casino industry in Las Vegas 30 years ago as a casino dealer, eventually spent 11 years on the casino floor as a supervisor and  pit manager before moving on to surveillance director roles at Tropicana Hotel and Casino, and MGM Grand Las Vegas – a major casino with a huge VCR surveillance solution.

n between jobs in big casinos in the U.S. Page came out to Australia for 12 months in 1995 and oversaw the installation of SKYCITY Darwin’s analog surveillance solution as director of security and surveillance. After returning to the U.S. to oversee the installation of the CCTV system at Greektown Casino  in Detroit and at the Planet Hollywood Resort and Casino in Las Vegas Page came back to Australia in February of this year.

Page’s deep-rooted casino experience gives him a vigorous operational perspective when it comes to video surveillance and this is important because it means he approaches his digital surveillance solution with extreme clear-sightedness. While admiring the slick new technology it’s obvious that for Page, operational efficiencies are the vital ingredient.

As Page explains, SKYCITY Darwin’s old CCTV system, like all the surveillance solutions he’d worked with, was end-to-end analog and supported by VCRs that needed to be constantly babied throughout the day.

According to Page, when digital first came out the cost was too high from his perspective but he says that now prices have dropped the technology is now viable for casinos.

“When I returned to SKYCITY Darwin the first question I asked was: What are we doing with the surveillance system?” Page explains. “I was told by management that they were planning to install a digital Dallmeier solution. I didn’t know much about the Dallmeier system but I looked at everything I could find on the internet and soon realized it was right for us.”

“A great thing about the Dallmeier system from my operational perspective is the fact you don’t have to work with video tapes and all the problems associated with them,” Page says.

“You don’t have to take operators away from their workstations for 30 or 40 minutes at a time to change out tapes. You don’t have to store and manage those tapes, you don’t have to incur the cost of purchase of new tapes and new VCRs year after year.”

“I couldn’t put a figure on the improvement the switch to digital has brought us. However from the point of view for staff productivity it’s huge,” he says. “Digital means more hours for staff to observe the monitors  and protect the assets of the casino. If you have 2 people at the monitors on a quiet day and one of them is off changing tapes for 30-40 minutes that’s a set of eyes taken away.”

Page says the intuitive archive management system is the best feature of the new digital Dallmeier solution in terms of day to day operations.

“In the old days if an incident occurred you’d have dig out the log books and locate  the tapes then some one would have to sit down and view the tapes,” Page says. “With this system you can search by date by time, by location by name, by whatever – there’s a drop down menu and the system will find the video for you.”

“And it’s redundant the way the system is designed – I know there’s no possibility of losing coverage – that was always a problem with tape”

Planning the solution


The decision to install a Dallmeier solution was driven by SKYCITY Darwin consultant Craig Graham of Radford Consulting and the casino’s surveillance manager, Greg Bretag. The system Craig and Greg selected combined the Dallmeier SeMSy security management system with Dallmeier’s DIS-2 MPEG2/MPEG4 dual streaming encoders with onboard RAID-1 storage.

This was an interesting choice and an astute one. A system with this structure not only gives a great deal of functionality, it’s also loaded with layers of redundancy at multiple levels. According to C.R. Kennedy’s Vlado Damjanovski, failure of one of the two drives in each encoder will not compromise the integrity of the recording.

“Each DIS-2 is hot-swappable and each subrack hosting 10 DIS-2 encoders runs on dual redundant power supplies with dual UPS support,” Vlado explains. “Even the Cisco network switchers are designed with redundancy in mind so that system uptime at SKYCITY Darwin is projected to hit the same numbers as Dallmeier’s Macau Casino-proven uptime with “six nines” at 99.99994 per cent – that’s higher than the best servers in the world.”

Vlado says that SKYCITY Darwin chose broadcast quality MPEG2 rather than the more common MPEG4. This compression protocol offered picture quality no different from the live image coming from the camera, irrespective if this was live of playback.

What’s important in this decision is the fact the architecture of the Dallmeier solution allowed the end user to choose a compression protocol dedicated to image quality, not optioned for streaming through skinny network pipes.

Integral to the installation and its commissioning was Steve Maynard, general manager of C.R. Kennedy’s CCTV Division. Steve and Vlado were heavily involved from start to finish with this job.

“When we won the job at the end of last year the first thing Vlado and I did was to come up to Darwin with an analyzer and made a record of the signals on every piece of surveillance cable in the casino,” explains Steve.

“We audited the system completely and ultimately that was the right approach. Surveillance in a casino is vital to the operation of tables so this analysis was central to ensuring the casino would always have the coverage it needed to legally operate during throughout the upgrade.”

According to Maynard, most the work conducted by C.R. Kennedy and ACE Cabling took place behind the patch panel into which all the system’s cameras are directed on their way to the server room.

“Cameras in the field come back by RG-59 to the patch panel and then go onto thin coax to the server room and into the encoders,” Steve says. “There are 360 inputs at the patch panel and 360 inputs with expansion space for an additional 180 inputs down in the server room.”

“The server room is the heart of the SKYCITY Darwin installation and a stunning piece it is. Along with the fundamental cleverness of Dallmeier’s system design and a sweet general layout incorporating under-floor cabling, dual air conditioning units and integrated fire detection, the stand-out feature of the server room is the high quality of the cabling and looming. It’s simply beautiful work”


“Signals go from the encoders to the hard drives within the DIS-2 itself and they can then be accessed from anywhere on the network, with incidents stored to the archive server’s RAID-5 storage system if they are to be retained long term.”

“Live footage can be called up directly from the DIS-2 encoders from the control room when required but multiple live streams are not supplied there unless they are called for,” explains Steve. “The control room’s access to image streams is via a standard Fast Ethernet network. It’s a dedicated surveillance network and not connected to the casino’s data network at all.”

“From a network design point of view the system is relatively simple,” Steve says. “It’s a dedicated Fast Ethernet LAN that runs from the server room to the control room with no external ports – a fairly typical configuration for high security CCTV LANs.”

“There’s an access layer controlled by core servers,” Steve says. “These are all Cisco switchers – 2960 Series access switchers for the access layer and 3560 series core switchers handling the core.”

“Each of these core switchers is connected to each access layer switch. There are 8 access layer switches, each 48 port, with the core switches being 24 port,” Steve explains. “With this installation you can lose a whole core switch and still have plenty of workstations operational.”

“The idea is that no matter what happens up here the guys have is something that’s fully redundant and very robust – under all circumstances the system will keep delivering,” Steve explains. 

“From a technical standpoint something that’s really important is that the architecture of this system is specifically designed so the surveillance network runs on layer 2 with SNOOP and without the need for a gigabit LAN,” Steve explains.

And Steve says the Dallmeier system has multicasting to reduce data traffic when multiple operators select the same live view.

“With many other digital surveillance solutions the encoders transfer all the data from the cameras onto the network all the time but in this system, the data is encoded to HDDs within the DIS-2 and does not go directly onto the network unless it’s requested,” he explains.

“So the big difference here is that all the bandwidth heavy work is being done off the network. If video was being transferred directly onto the network then you’d need Layer 3 support – and at Layer 3 the cost in network infrastructure becomes enormous.”

“Very important is that this overall system configuration as installed at SKYCITY Darwin allows you to build very large digital solutions – the largest digital system in the world uses Dallmeier’s system architecture with more than 3500 cameras,” explains Steve.

Operational capability

No account of this solution is complete without recognition of the capability of Dallmeier’s SeMSy video management software. The control platform is extremely hard to disassociate from the Dallmeier’s digital matrix and in some ways it’s best think of them as a single entity.

The digital matrix delivers full frame resolution and real time storage for every channel, with images able to be recorded from any camera and simultaneously switched to a linked monitor using SeMSy. Operators use SeMSy to view images in live and playback mode, while recording continues simultaneously.

“From an operational point of view the reason this Dallmeier system uses MPEG-2 as opposed to MPEG-4 is that MPEG-2 offers better quality,” says Steve. “MPEG-2 is also a broadcast standard and able to be played on almost any device.”

“In a casino you simply have to see fine detail – a ball falling into a hole, a card, a dice,” he explains. “MPEG-4 won’t give you the clarity that means such a lot to a casino.”

All the cameras are being recorded in full frame MPEG2 at 25 frames per second and the SKYCITY Darwin guys have chosen 3 megabits/s per channel.

“Regardless of this, Ethernet requirements are still minimal because the cameras are being recorded off the network – that’s very important,” Steve explains.

“The workstations only need 100Base but a camera still has the potential to be viewed 9 x 4 x 6 x 3 times, plus whatever multiscreens are being deployed.”

Allowing the operators to display multiple screens in 9 or 16-way spilts on large screen formats, Dallmeier Vario-Decoders at SKYCITY Darwin utilise a Gigabit portion of the CCTV LAN.

“The Vario-Decoders handle multiscreen decoding in full frame MPEG2 real time for 4-way, 9-way splits, and do 16-way splits in nearly real time.

“This is in addition to the 6 operators operator each one having three true composite monitors in real time” Steve says. “These big configurations allows all operators to keep an eye on key areas and call up incidents and review playbacks on their own workstations or system keyboards.”

Another interesting characteristic of this solution is the fact there are some CRT monitors in the control room along with the plasma screens and LCD screens.

“It was felt in the selection process that good quality CRT gave superior viewing quality compared to LCD at these full frame MPEG2 resolutions, simple as that,” says Bill. “Overall picture quality was just better and CRT has better reliability.”

Meanwhile, supporting mouse operation of the GUI is an external keyboard with tons of onboard processing power that’s able to switch the matrix, review, archive, control PTZs and activate specific functions. A jog/shuttle is integral part of the Dallmeier VMC keyboard and it offers preset keys to switch between live and playback instantly. Keyboard functions include one button to search, another to archive, one button to burn to DVD – then there’s jog shuttle, frame by frame search and easy manipulation.

According to Steve the system keyboard has a lot of program capability and this makes it ideal for casinos.

“The system can operate differently using keyboards with onboard processing that take shortcuts to give operators faster reaction time, or it can be done via graphical interfaces on workstations using a mouse,” Steve explains.

“There are maps, there’s a lot of feedback to the operator and there’s a warning system integrated into the system that allows it to report things like HDD failure.” 

“All the maps are original AutoCAD maps and can be one touch loaded to the server and then downloaded from there to the workstations,” he says. “You can select a camera directly from a map same as one touch selection on the keyboard. Maps will follow the camera selection.  You can edit maps, add cameras and numbers directly into the AutoCAD drawing and the system will recognize these automatically. This has never been done in CCTV before.”

Bill agrees the Dallmeier system with its SeMSy video management system is extremely easy for staff to operate.

“It’s simple to operate as well as having all the bells and whistles you could possibly ever want,” Bill says. “It allows surveillance staff much greater access to the video footage, allowing them to not only monitor an area more effectively but to better support security staff on the floor. ”

“That’s what you get with digital. We can track a wide range of situations across the casino floor, we can have point of sale interfacing allowing us to track data from that source as well. There are all sorts of analytical tools available. The Dallmeier system is great.” 

Bill says that from his operational point of view, the archive management system is the best element of the SyMSy video management solution.

“In the old days if an incident occurred you’d have dig out the log books and locate the tapes then some one would have to sit down and view the tapes. With this system you can search by date by time, by location by name, by whatever – there’s a drop down menu and the system will find the video for you,” Bill says.

“And because the system is designed to be redundant at multiple levels I know there’s no possibility of losing the coverage.” Some of the operators were nervous making the change from analog and thought they might have trouble operating the system but it’s so easy they’ve adapted.”

Bill says that while the decision to go with Dalmeier was made before he came on board, it was based on the fact this solution was being used in major casinos around the world.

“Importantly, with the Dallmeier system we could send people to see the same system working perfectly in applications much larger than our own like the large casinos in Macau.” Bill explains. “We were obviously interested in latency but we found it was very minimal operationally.”

Steve says latency is the big question that arises with digital solutions but he says latency with the Dallmeier solution is a non issue.

“The system is very user friendly in that regard and obviously that was one of the key performance criteria with the SKYCITY Darwin system,” he says. “With viewing of single cameras from single encoders there’s virtually no latency – maybe a hundred milliseconds.”

“The latency and the control issues were question marks and they have been well and truly answered,” Steve explains. “And the quality of this system versus the hybrid solutions available made this a much better option.”

“I think a lot of people have a false sense of security in relation to their analog hardware – it’s a big primary piece of gear to go down – you lose a switcher and that’s the end of that,” Steve says. “The matrix switcher at SKYCITY Darwin was 12 years old and becoming hard to maintain – the level of redundancy is greater now, and at all levels.”

Steve says other strengths of the Dallmeier system include the fact digital has a lot a capability for integration including support for things like Micros cash management systems, as well as facial recognition and numberplate recognition.

“Another great strength of this solution is a mobile disaster recovery module that can be handled at a single workstation via a laptop,” he says. “It’s possible for the casino to retain control of the system under the most adverse and unlikely conditions. Mind you, recording never stops irrespective of how you control the system.”

“I can’t imagine a normal situation this solution is not equipped to handle with ease,” Steve says. “But if anything comes up, the control room will still be in charge of surveillance.”

According to Bill Page, while digital has a higher up front cost than analog solutions he says the maintenance is significantly less and the functionality vastly greater.

“From an operational point of view the key thing about this Dallmeier system is the user friendly and powerful archive management – it’s perfect for operators viewing and recalling incidents,” Page says.

“There are the bells and whistles but that’s the heart of the system for me. You could save a tape for 5 years and then view it again and possibly have a VCR eat the tape. When an incident occurred you had to save the original tape then you had to make a copy of the incident, then you had to put it all on a master tape so there were 3 VHS tapes for each incident. In the event of an incident you quadrupled the number of tapes you had to manage.”

Bill says the modularity of the system makes easy to maintain, flexible, easy to expand and easy to upgrade, too. 

“The system has the potential to integrate with software analytics that look at card counting and so on. We’ve got no plans for this yet but it’s nice to know the system is capable of supporting that software in the future. It is also capable of integration with the access control system.”

 Steve agrees.

“The access system here is Concept and Inner Range people have made a great interface with Dallmeier so it would be very easy to integrate Insight with this solution at some point in the future,” he explains.

There are a bunch of casinos in Australia using Dallmeier products – some have done so for more than a decade with most adding digital components to their existing analogue matrices. Such casinos include Burswood in Perth, Crown in Melbourne, Reef in Cairns, Lasseters in Alice Springs. But this new solution at SKYCITY Darwin takes Dallmeier’s engineering to a new level.

The installation


The Dallmeier installation at SKYCITY Darwin was characterized by SKYCITY Darwin’s request to make the switch to digital with no downtime. According to Vlado, this was a vital request. No casino in Australia can operate without a CCTV system running and that means surveillance is integral to casino operation – no cameras, no business.

Vlado says that to the satisfaction of the customer, C.R. Kennedy and Ace Cabling delivered this.

“The project was completed in record time just as the consultant and customer required, making this project a show piece of what can be done with good planning and organisation.”

Same as most large installations, the SKYCITY Darwin installation called for a layered approach with multiple stages. Firstly, a temporary new Dallmeier system was installed in half of the SKYCITY Darwin human resources training room and then operators were trained in its use. At this stage, both, analogue and digital matrices, were running in parallel.

“Once the operators were up to speed, they moved to the temporary system and the old system was shut down and the equipment and cabling pulled out of the old spaces,” says Steve. “The new control room was then built and when it was completed, the team moved from the temporary control room to the new one – all this was well planned and went by the numbers.”

Steve says that overall there were a couple of little hiccups relating to equipment delivery, often a problem fro Darwin, but that’s about all.

“During the early stages of the project we were delayed because the racks for the server room were bent while in transit and we had to address that problem,” he explains.

“Strongabuilt manufactured some new parts quickly and we were able to resuscitate other parts.”

“You see some of the key characteristics of the Dallmeier solution in the server room at this casino,” Steve says.

“Racks and rack  cable management systems were designed specifically for this application by Kevin Wilson from Strongabuilt. 

“Racks are built to be serviceable front and back and the system displays a high level of redundancy. It’s a very reliable solution and that comes from experience with big casino installations – big ones like Macau.”

The server room is the heart of the SKYCITY Darwin installation and a stunning piece it is. Along with the fundamental cleverness of Dallmeier’s system design and a sweet general layout incorporating under-floor cabling, dual air conditioning units and integrated fire detection, the stand-out feature of the server room is the high quality of the cabling and looming. It’s simply beautiful work.

What throws the excellence of these looms into such sharp relief is the rudimentary standards evident in older parts of the cable plant. Like any elderly building, SKYCITY Darwin has been worked on by a large number of contractors over a very long period of time and the result is a sprawling cabling infrastructure of variable quality and unusual configuration.

“Handling the legacy cabling was a major task,” explains Bill. “It was literally spaghetti. Each individual cable had to be isolated and tested to establish its content and once that was done it was either tagged or chopped, depending on what the testing found.”

To some extent replacing and upgrading the existing cable plant became a defining feature of the installation, with a large amount of testing and tagging required to ensure the system was robust enough to meet operational standards as various key times throughout changeover process.

According to Steve, even though the site is not a large one, ACE Cabling still put one-and-a-half tons of RG-59 in a cable management system in the ceiling void above the casino floor with a combined distance of about 20km. 

Bill says a particular issue during the installation was the discovery that the old style PTZs which use a DTMF tone for data comms were not compatible with the new system.

“We had to remove dozens of these cameras and replace them – then power supplies became an issue because multiple of these old PTZs were daisy chained to a single power supply,” he explains. “That had to be cleaned up, too.”

Bill says that an integral part of the job was meeting with upper management to keep them informed of how the work was progressing.

“Surveillance manager Greg Bretag and I would meet weekly with upper management to inform them where we were at and to outline any risks and requirements,” explains Bill.

“It was important that management knew exactly where we were throughout the upgrade,” he says. “SKYCITY Darwin management was very involved and supportive – that was important to the success of this integration.”

Bill says an integral part of the installation and commissioning process involved training operators to manage the new system.

“The training undertaken during installation went very well,” Bill says. “We broke it down into technicians and operators and then went through a schedule – the system was not really all up and running at that point but it went well.”

“As part of that process we created our own training manual in addition to the Dallmeier training manual – we gave that manual to new operators when they begin. As we add functionality we add that in to the manual,” he says.

Steve agrees.

“Taking 20 operators and training them all on an entirely new system, while the old system is still running – it’s harder than you think,” he says.

Another important element of this installation was customization of the video management software.

“The archive management system is also customized during installation,” says Bill. “It’s a matter of sitting down with the system designers and integrators and letting them know what functions are needed and if this can be accomplished  within the guidelines of the system.”

Typically all SeMSy video management solutions get some customization, according to Steve.

“There’s a process in the installation where elements of customization can be included in the system design – frame rate, resolution and so on,” Steve explains.

“Some casinos like more latency and higher image quality - it depends on the applications. Then there are things like Micros point-of-sale integration or fine tuning of the Vario-Encoders.”

According to Bill, when the new control room was finished, the temporary control room was switched over.

“We switched over at about 4.30 in the morning – we had 3 teams of 2 technicians and they went around and switched the data for the PTZs to make ensrue that the  controls of the cameras were operational – that went really well,” Bill explains.

“We had worked out spreadsheets that assessed every camera and considered what day they could be switched over right down to the time of day. Our temporary control room was a case in point. We had to be out of there by a certain date and we met that date – it was very pleasing.”

Steve agrees.

“We hit all the right targets and the system even came in a little bit under budget – that was popular,” Steve says. 

 “The final commissioning and programming was done by two experienced engineers from Dallmeier; Umberto  Vicenzotto and Philip Rissettio; assisted by Trevor Joseph from CRK Queensland office, and Paul Cafeo from CRK Melbourne office,” Steve explains.

“Those engineers came all the way from Germany and they put in some long hours – I thought that was fantastic,” Bill says. And Bill clearly felt Dallmeier’s engineering was enhanced by the support of C.R. Kennedy’s dedicated people.

“An important part of this system is maintenance and even though the system is now installed and the contract completed, C.R. Kennedy still comes and views the system,” Bill says.

“With a casino it’s important to keep the system working perfectly – that’s what CR Kennedy gives us. Other vendors were not as forthcoming in terms of system support but these guys made extra effort during the installation and after it.”

Steve says C.R. Kennedy is delighted with the system, too.

“From C.R. Kennedy’s perspective we’re obviously very proud of this installation,” he says. “We think SKYCITY Darwin is the first casino in Australia, and probably in the world, that had an existing analogue matrix system converted to complete digital matrix, without any down time during the conversion.”

“Converting large analogue CCTV systems into digital solutions is not only possible but easy to do - that's what we have proven with the SKYCITY Darwin project,” Steve says.

 

“From C.R. Kennedy’s perspective we’re obviously very proud of this installation. We think SKYCITY Darwin is the first casino in Australia, and probably in the world, that had an existing analogue matrix system converted to complete digital matrix, without any down time during the conversion”