SYS-CON MEDIA Authors: Pat Romanski, Gary Arora, Zakia Bouachraoui, Yeshim Deniz, Liz McMillan

Blog Feed Post

HTTP Request Cloning via iRules, Part 1

One of the requests that I've seen several times over the years is the ability to completely clone web requests across multiple servers. The idea is that you can take the HTTP traffic coming in bound for pool member A and distribute it, in its entirety, to pool member B. Or perhaps members B-G..whatever your needs are. This can be helpful for many reasons, security auditing, test or dev harnesses, archival, etc. Whatever the reasons, this has been a repeated question in the forums and in the field. While clone pool functionality works to some degree for this, it doesn't work quite as desired, and doesn't easily distribute to multiple additional members.

iRules, however, offers a solution.

Using the HSL feature in iRules that, if you remember, allows you to specify a protocol and destination, which can be a pool, we are able to treat this much like sideband connections in v11. By establishing a new connection and sending across the HTTP info as needed we're able to clone the HTTP traffic in its entirety. Let's take a look at how this starts:

   1: when CLIENT_ACCEPTED {
   2:     # Open a new HSL connection if one is not available
   3:     set hsl [HSL::open -proto TCP -pool http_clone_pool]
   4:     log local0. "[IP::client_addr]:[TCP::client_port]: New hsl: $hsl"
   5: }

As you can see, it's straight-forward enough. Using the HSL::open command we set the protocol to TCP and the pool to whichever pool you'd like to clone your HTTP traffic to. Now that we know where and how we're sending the data, we need to figure out which data to send. The only trick with HTTP in this step is that GET and POST requests need to be handled differently. With a POST we will need to collect the data that is being posted so that we can replay it back to the new destination. With a GET we simply forward through the headers of the request. Fortunately determining which is which is a cake walk in iRules, so it's just the collecting and forwarding bit we really need to worry about. This is the real "meat" of this iRule, and even that isn't difficult, it looks like:

   1: when HTTP_REQUEST {
   2:     # Insert an XFF header if one is not inserted already
   3:     # So the client IP can be tracked for the duplicated traffic
   4:     HTTP::header insert X-Forwarded-For [IP::client_addr]
   5:  
   6:     # Check for POST requests
   7:     if {[HTTP::method] eq "POST"}{
   8:         
   9:         # Check for Content-Length between 1b and 1Mb
  10:         if { [HTTP::header Content-Length] >= 1 && [HTTP::header Content-Length] < 1048576 }{
  11:             HTTP::collect [HTTP::header Content-Length]
  12:         } elseif {[HTTP::header Content-Length] == 0}{
  13:             # POST with 0 content-length, so just send the headers
  14:             HSL::send $hsl [HTTP::request]
  15:             log local0. "[IP::client_addr]:[TCP::client_port]: Sending [HTTP::request]"
  16:         }
  17:     } else {
  18:         # Request with no payload, so send just the HTTP headers to the clone pool
  19:         HSL::send $hsl [HTTP::request]
  20:         log local0. "[IP::client_addr]:[TCP::client_port]: Sending [HTTP::request]"
  21:     }
  22: }

As you can see this is pretty standard iRules fare for the most part. HTTP::method, HTTP::header, HTTP::collect. Nothing shocking for the most part. The real trick is in the HSL::send command. Note that it's going to "$hs1"? That's the connection we established earlier with the HSL::open command. Now that we have that handle available we're able to easily forward through other traffic. So as you can see in the POSTs with content attached  we're collecting, and anything else we're forwarding along the headers alone. Note that nothing has been sent for the POSTs that have content attached, we've just entered a collect state so the client will continue sending data and we'll store it. That data is then available in the HTTP_REQUEST_DATA event, and we can forward it along when that occurs.  So for those particular requests an additional event will fire:

   1: when HTTP_REQUEST_DATA {
   2:     # The parser does not allow HTTP::request in this event, but it works
   3:     set request_cmd "HTTP::request"
   4:     log local0. "[IP::client_addr]:[TCP::client_port]: Collected [HTTP::payload length] bytes,\
   5:         sending [expr {]string length [eval $request_cmd][ + ]HTTP::payload length[}] bytes total"
   6:     HSL::send $hsl "[eval $request_cmd][HTTP::payload]"
   7: }

Now that the HTTP_REQUEST_DATA event has fired we know our collect has picked up the data we want it to. This event will only fire after a successful HTTP::collect. Once this happens we're ready to forward along the POST and the accompanying data.  After a little expr trickery to convince the parser to allow the HTTP::request command within the HTTP_REQUEST_DATA event (it doesn't think it should work, but it does...so we trick it) we're able to send along the original request and payload data without much hassle. Again making use of the HSL::send command and the $hs1 variable we set up at the beginning makes this process easy.

At this point you now have a functioning iRule that will clone traffic inbound for your Virtual to another pool of your choosing. At this point you are probably asking yourself three questions.

1) Why hasn't this been written before?

2) Where is the version that allows forwarding to multiple other pools?!

3) Why HSL and not sideband connections?

Well, those answers are simple:

1) Because our good friend Hoolio hadn't written it yet! Aaron whipped this together and posted it. I got his okay to write it up and get it out there, so here it is. Keep in mind that this is VERY early in the testing stages and is prone to change/update. I'm sharing it here because I think it's awesome, and don't want it to slip off into the night without being called out. But this is very much a use at your own risk sort of thing for now. I'll update with notes when more testing has been done. Also worth note is that this requires at least version 10.1 or newer to function.

2) It's coming, don't fret. That will be Part 2! You did notice the Part1 in the title didn't you? We can't give it all away at once. Besides that part is still under testing. Releasing it before it's ready wouldn't be prudent. Stay tuned, it's coming.

3) I asked Aaron the exact same thing and here's what he said:

HSL automatically ACKs the server responses, but ignores the data.  From limited testing of both HSL and sideband connections, HSL is also a lot more efficient in handling high connection rates.  Also, HSL is available from 10.1 and sideband only on 11.x.

So there you have it. Sideband connections would work just fine, but HSL allows for a wider audience (10.1 and above), and offers a little added efficiency/ease of use in this particular case. Keep in mind that HSL won't handle many of the more complex scenarios that sideband connections will, hence the tradeoff, but in this particular case HSL seems to win out.

Keep an eye out for the next installment of this two-parter, wherein you'll see how to extend this model to work with multiple clone destinations. For now, any questions/comments are welcome. If you get this up and running in test/dev, please let us know as we'd love any feedback resulting from that testing. Many thanks to hoolio for his continued, outstanding contributions. With that, I'll leave you with the full version of the iRule:

   1: when CLIENT_ACCEPTED {
   2:     # Open a new HSL connection if one is not available
   3:     set hsl [HSL::open -proto TCP -pool http_clone_pool]
   4:     log local0. "[IP::client_addr]:[TCP::client_port]: New hsl: $hsl"
   5: }
   6: when HTTP_REQUEST {
   7:  
   8:     # Insert an XFF header if one is not inserted already
   9:     # So the client IP can be tracked for the duplicated traffic
  10:     HTTP::header insert X-Forwarded-For [IP::client_addr]
  11:  
  12:     # Check for POST requests
  13:     if {[HTTP::method] eq "POST"}{
  14:         
  15:         # Check for Content-Length between 1b and 1Mb
  16:         if { [HTTP::header Content-Length] >= 1 && [HTTP::header Content-Length] < 1048576 }{
  17:             HTTP::collect [HTTP::header Content-Length]
  18:         } elseif {[HTTP::header Content-Length] == 0}{
  19:             # POST with 0 content-length, so just send the headers
  20:             HSL::send $hsl [HTTP::request]
  21:             log local0. "[IP::client_addr]:[TCP::client_port]: Sending [HTTP::request]"
  22:         }
  23:     } else {
  24:         # Request with no payload, so send just the HTTP headers to the clone pool
  25:         HSL::send $hsl [HTTP::request]
  26:         log local0. "[IP::client_addr]:[TCP::client_port]: Sending [HTTP::request]"
  27:     }
  28: }
  29: when HTTP_REQUEST_DATA {
  30:     # The parser does not allow HTTP::request in this event, but it works
  31:     set request_cmd "HTTP::request"
  32:     log local0. "[IP::client_addr]:[TCP::client_port]: Collected [HTTP::payload length] bytes,\
  33:         sending [expr {]string length [eval $request_cmd][ + ]HTTP::payload length[}] bytes total"
  34:     HSL::send $hsl "[eval $request_cmd][HTTP::payload]"
  35: }

Read the original blog entry...

More Stories By Colin Walker

Coming from a *Nix Software Engineering background, Colin is no stranger to long hours of coding, testing and deployment. His personal experiences such as on-stage performance and the like have helped to foster the evangelist in him. These days he splits his time between coding, technical writing and evangalism. He can be found on the road to just about anywhere to preach the good word about ADCs, Application Aware networking, Network Side Scripting and geekery in general to anyone that will listen.

Colin currently helps manage and maintain DevCentral (http://devcentral.f5.com). He is also a contributor in many ways, from Articles to Videos to numerous forum posts, to iRules coding and whatever else he can get his hands on that might benefit the community and allow it to continue to grow.

Latest Stories
Every organization is facing their own Digital Transformation as they attempt to stay ahead of the competition, or worse, just keep up. Each new opportunity, whether embracing machine learning, IoT, or a cloud migration, seems to bring new development, deployment, and management models. The results are more diverse and federated computing models than any time in our history.
On-premise or off, you have powerful tools available to maximize the value of your infrastructure and you demand more visibility and operational control. Fortunately, data center management tools keep a vigil on memory contestation, power, thermal consumption, server health, and utilization, allowing better control no matter your cloud's shape. In this session, learn how Intel software tools enable real-time monitoring and precise management to lower operational costs and optimize infrastructure...
"Calligo is a cloud service provider with data privacy at the heart of what we do. We are a typical Infrastructure as a Service cloud provider but it's been designed around data privacy," explained Julian Box, CEO and co-founder of Calligo, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
Isomorphic Software is the global leader in high-end, web-based business applications. We develop, market, and support the SmartClient & Smart GWT HTML5/Ajax platform, combining the productivity and performance of traditional desktop software with the simplicity and reach of the open web. With staff in 10 timezones, Isomorphic provides a global network of services related to our technology, with offerings ranging from turnkey application development to SLA-backed enterprise support. Leadin...
While a hybrid cloud can ease that transition, designing and deploy that hybrid cloud still offers challenges for organizations concerned about lack of available cloud skillsets within their organization. Managed service providers offer a unique opportunity to fill those gaps and get organizations of all sizes on a hybrid cloud that meets their comfort level, while delivering enhanced benefits for cost, efficiency, agility, mobility, and elasticity.
DevOps has long focused on reinventing the SDLC (e.g. with CI/CD, ARA, pipeline automation etc.), while reinvention of IT Ops has lagged. However, new approaches like Site Reliability Engineering, Observability, Containerization, Operations Analytics, and ML/AI are driving a resurgence of IT Ops. In this session our expert panel will focus on how these new ideas are [putting the Ops back in DevOps orbringing modern IT Ops to DevOps].
Darktrace is the world's leading AI company for cyber security. Created by mathematicians from the University of Cambridge, Darktrace's Enterprise Immune System is the first non-consumer application of machine learning to work at scale, across all network types, from physical, virtualized, and cloud, through to IoT and industrial control systems. Installed as a self-configuring cyber defense platform, Darktrace continuously learns what is ‘normal' for all devices and users, updating its understa...
Enterprises are striving to become digital businesses for differentiated innovation and customer-centricity. Traditionally, they focused on digitizing processes and paper workflow. To be a disruptor and compete against new players, they need to gain insight into business data and innovate at scale. Cloud and cognitive technologies can help them leverage hidden data in SAP/ERP systems to fuel their businesses to accelerate digital transformation success.
Most organizations are awash today in data and IT systems, yet they're still struggling mightily to use these invaluable assets to meet the rising demand for new digital solutions and customer experiences that drive innovation and growth. What's lacking are potent and effective ways to rapidly combine together on-premises IT and the numerous commercial clouds that the average organization has in place today into effective new business solutions.
Concerns about security, downtime and latency, budgets, and general unfamiliarity with cloud technologies continue to create hesitation for many organizations that truly need to be developing a cloud strategy. Hybrid cloud solutions are helping to elevate those concerns by enabling the combination or orchestration of two or more platforms, including on-premise infrastructure, private clouds and/or third-party, public cloud services. This gives organizations more comfort to begin their digital tr...
Keeping an application running at scale can be a daunting task. When do you need to add more capacity? Larger databases? Additional servers? These questions get harder as the complexity of your application grows. Microservice based architectures and cloud-based dynamic infrastructures are technologies that help you keep your application running with high availability, even during times of extreme scaling. But real cloud success, at scale, requires much more than a basic lift-and-shift migrati...
David Friend is the co-founder and CEO of Wasabi, the hot cloud storage company that delivers fast, low-cost, and reliable cloud storage. Prior to Wasabi, David co-founded Carbonite, one of the world's leading cloud backup companies. A successful tech entrepreneur for more than 30 years, David got his start at ARP Instruments, a manufacturer of synthesizers for rock bands, where he worked with leading musicians of the day like Stevie Wonder, Pete Townsend of The Who, and Led Zeppelin. David has ...
Darktrace is the world's leading AI company for cyber security. Created by mathematicians from the University of Cambridge, Darktrace's Enterprise Immune System is the first non-consumer application of machine learning to work at scale, across all network types, from physical, virtualized, and cloud, through to IoT and industrial control systems. Installed as a self-configuring cyber defense platform, Darktrace continuously learns what is ‘normal' for all devices and users, updating its understa...
Dion Hinchcliffe is an internationally recognized digital expert, bestselling book author, frequent keynote speaker, analyst, futurist, and transformation expert based in Washington, DC. He is currently Chief Strategy Officer at the industry-leading digital strategy and online community solutions firm, 7Summits.
Addteq is a leader in providing business solutions to Enterprise clients. Addteq has been in the business for more than 10 years. Through the use of DevOps automation, Addteq strives on creating innovative solutions to solve business processes. Clients depend on Addteq to modernize the software delivery process by providing Atlassian solutions, create custom add-ons, conduct training, offer hosting, perform DevOps services, and provide overall support services.