Posts Tagged ‘MFT Agent’

Is your business getting indigestion? IBM MQ can ease that pain by ingesting your file data into MQ including MQ on Cloud.

January 14, 2019

egg-eating-snake-856

Christmas and New Year is over for another year. At this time of year, it can be easy to eat and drink too much. Consuming too much can lead to indigestion and the results can be unpleasant.

 

But have you considered it might be similar when you move data through your enterprise? Data can be large. Data can be small. But once it exists, it has a purpose. And that means it has a use, and value. In that case it should delivered, in a timely way, with security and reliability, to where it can add value to the business.

 

However, moving the data can be a problem. Data can be moved by the application as it is created. And certainly, IBM MQ has a long history of being an ideal solution for this, as it is designed to connect applications, exchanging data reliably and asynchronously.

MQ messages have a maximum size of 100MB. Which is actually very large for individual application generated messages, especially if you are sending the data out as it is created, so while some use cases do use very large message sizes, mostly it is much smaller. And not only is MQ optimized for this traffic, enabling it to send millions, or even billions of messages per day through your network, your own infrastructure is likely built to meet this need.

 

But consider when data is created, or pulled in from elsewhere, and may be at rest in the filing system. It needs to move through the business to where it will add value. But this data in the filing system might be thousands, or millions of individual records, imported or built up over time. Trying to send gigabytes, or even terabytes of data in one lump is going to give your network the equivalent of indigestion. It’s going to be blocked up until it can pass through. Traditional file transfer approaches suffer from this issue.

 

Let’s think of how this might happen. You are a retailer. Some of your stores process their transactions as they happen, flying through the network as each one is small. Others instead batch them up and send them as a file. The file can be very large, and if coming from a remote location could take minutes or even hours to come through the network, because of the way that networks can slow down data transfer rates because of errors. This impacts the ability of the business to act on this data.

wmqfte_intro

 

An important feature of MQ Advanced (and MQ Appliance), and now MQ on Cloud is the ability to ‘ingest’ the data from files on the file system into MQ. This data is then moved as MQ messages through your network. As even the largest files are automatically broken into chunks suitable for sending as MQ messages, with all the reliability, security and assured delivery that MQ provides, your business gets the benefit of the data delivery, without suffering ill effects from the movement.

 

Moving all data, from applications and the file system, all through a single reliable high-performance pipe like IBM MQ gives your business the assurance that all data is handled with the right care and attention. And your business suffers no ill effects even when handling the biggest inputs. Allowing more of the data traffic to move rapidly and reliably through your network, without everything slowing down.

Your data is no longer getting stuck in a file, or in a remote system. It won’t even get lost moving between systems. It is moving freely between systems as it moves as MQ messages. No single message is too large for the network. And the business gets to benefit from your data now being handled and processed directly as MQ messages. It is no longer file data, so no longer stuck in the slow lane. Data ingestion is better than indigestion. Accelerate your data use by ingesting your file data with MQ on Cloud, MQ Advanced or MQ Appliance.

Don’t forget you can download and try all the features of MQ Advanced for free from this download page and you can also try MQ on Cloud in just a few clicks here.

screenshot 2019-01-14 at 18.44.43

Advertisements

Data in motion is data adding value. Using MQ for data transfer from files as well as applications

September 21, 2018

gold bar

Data. It’s sometimes described as the new gold. Certainly, it is valuable. But how exactly does it compare to gold?

The value of gold is that it can be made into gold objects such as jewelry which will have both the value of the metal and have additional value because it has been put to higher value use. But there are also costs associated with owning gold, keeping it safe, and moving it safely and reliably from where it is to where it needs to be. In order to realize the value, the gold jewelry must be moved from where it has been created to where it is needed.

 

What about data? Like a delicate piece of gold jewelry, some people might value a piece of data highly, but to others it is of little or no value. While gold is essentially fungible – as any gold can be fashioned into something, each piece of data is unique, not just in itself but set in its own context. Think of when you are trying to complete on a purchase of a house. An agreed mortgage is all very well, but what’s important is that the transfer of funds happens at a specific time to a specific account to allow the purchase to go through.

 

While a piece of data might represent something, the value of the data is only really achieved when it is moved to the right place at the right time. Having the data held securely in a system is not valuable on its own. It needs to be moved to be valuable. Data is created somewhere in the infrastructure but needs to be consumed somewhere else to add value.

 

This gives us a number of problems with data. Storing it safely; moving it safely; knowing it has arrived, so it can be put to use. This is again similar to gold. You have to keep it safe, and certainly you have to look after it as it moves as well. And if you have made a piece of jewelry for someone, you have to let them know you have it ready for them.

 

In your business you will have huge quantities of data. Data that is being created every second and stored in your file system. Buried treasure. I have already discussed this in here before. So what are you going to do?  You need to move it safely and securely to the part of the business that can make use of it. Probably the faster you move it, the better. And you certainly don’t want to lose it. Or have it stolen as it moves. And wouldn’t it be helpful if as it was delivered to the destination, the target application could immediately be made aware of it.

 

All this of course is handled for you by the Managed File Transfer component of MQ Advanced or MQ Appliance. The name is slightly misleading as it doesn’t move files but instead the contents of files can be moved as MQ messages. This means they take advantage of MQ’s unmatched secure and reliable delivery. No lost messages. No lost data. Your data gold will reach where it needs to go. And it can even be delivered directly to the target application without being written to the file system again.

 

Just as you wouldn’t want to keep all your gold in a vault where it wouldn’t add value to your business, don’t keep your data held up not adding value. Put it to work by moving it through MQ. When you have MQ Advanced or MQ Appliance you can deploy unlimited numbers of MFT Agents inside or outside your business. You can even embed them inside applications you share with your partners or suppliers. And the latest updates in MQ V9.1 add further enhancements to the MFT functions.

treasure

Think of all the buried treasure that has been lost over the years. Don’t let your data join those wasted resources. Get started today by learning MQ or with downloading a MQ trial. Or see what you can do with a hosted MQ on IBM Cloud or now on Amazon Web Services.

 

Not all data is treasure of course. You have to understand what’s valuable. But if it is valuable, then you should ask yourself why you aren’t moving it reliably through MQ Advanced, taking advantage of end-to-end encryption. After all you don’t want to go on a pointless quest with nothing at the end of it. Or find that your treasure has been taken.

journey_cost

Buried Treasure – embedding IBM MQ clients and MFT Agents into applications

June 13, 2017

treasure

I haven’t been doing this blog so long that I am going to repeat myself. Or at least not yet. But last year I did a blog on why you would use MQ – and that is broadly the topic of this entry as well but it comes from a specific use-case perspective. Plus – warning – it is longer than usual – sorry. Why do businesses, in their thousands, use IBM MQ – and its many different yet critical functions? Sadly, and I say this as the Offering/Product Manager for MQ, no one wakes up in the morning and decides they want to buy more IBM MQ – but they do so because of the benefit using MQ provides for the applications that run their business.

 

IBM MQ enables the exchange of data between applications, systems, services and files with reliability and security. It does this with scalability and simplicity. It has proved itself in doing this over the last 20+ years that much of the modern online business world takes IBM MQ, and its capabilities for granted.

 

The IT infrastructure is evolving rapidly – as it always is. As such there is both growth in new applications and existing applications are being updated and enhanced. Today’s applications typically have to be more resilient than ever, but also more portable – to be deployed pretty much anywhere. In most businesses applications will be extended out to business partners as the wider ecosystem is more tightly integrated than ever before.

 

These changes drive a greater need for seamless connectivity throughout the infrastructure and it makes it more important that all business data can be simply and quickly moved inside and outside the business. So how has IBM been working on IBM MQ to enable this? And will IBM MQ be able to help all customers – whether they are trying to connect and exchange data between applications, systems, services and files – not just the latest and greatest APIs?

 

IBM MQ allows for connectivity and exchange of data through MQ Clients and MQ MFT Agents and to make it easier for these to be used in many different use cases, IBM has been making changes to the packaging and licensing of these.

MFT Agents

One of the key changes was at the end of 2015, there was an update to the license documentation to allow for the redistribution of MQ Clients. IBM makes the MQ client libraries available for free download. These are then built into the MQ enabled applications to allow these applications to send and receive MQ messages. There is no cost for the MQ Clients – as they require a licensed MQ Queue Managers in order to function. However, until late 2015, the license prevent redistribution of these MQ Client files. This meant that if a business built the MQ Clients into an application, it wasn’t permitted to then distribute this application outside the business – i.e. it couldn’t share it with a business partner to allow that partner to work closely as an integrated partner. To allow this under the terms, the partner would need to either install the MQ Client library themselves or agree licensing terms to redistribute the MQ Client with IBM. This restriction was not helpful to these businesses or to the IBM MQ business and therefore it was changed to allow redistribution.

 

Now let’s look at a scenario – Company A uses MQ to exchange information throughout its business. It has suppliers (Company B and Company C) and it wants to streamline the manufacturing processes to enable them to get production statistics and thus help to plan for more efficient resupplies to their factories and warehouses. To do this it wants to provide them with a copy of their own in-house written application that uses MQ. Now that IBM allows for redistribution of the MQ Clients, Company A can simply provide their application to the partner companies to enable them to communicate seamlessly with no need to even be aware of the MQ Client embedded within the application. MQ messages can flow securely between the companies – and as only Company A has a MQ Queue Manager, they are the only ones licensed for MQ – and there is no additional MQ cost for this configuration. Note that companies exchanging MQ messages like this might want to make use of the MQ Internet Pass-thru feature to simplify passing messaging through their firewalls.

 

Now let’s imagine Company D. They are also part of the supply chain ecosystem for company A, and also many other businesses. But the stock control and distribution management systems are built mainly on files and file data. They keep these files updated with stock quantities and prices, but they find it simpler to keep using this method rather than online application updates and exchanges. They are used to sending these files to their customers using FTP but they always have a number of issues around FTP failures, reliability issues, and having to spend time diagnosing the problems inherent in these transfers.

 

Company A have a solution – the Managed File Transfer capability that is a part of IBM MQ Advanced. In place of regular FTP, the data inside the files can be sent as MQ messages from Company D to Company A, taking advantage of MQ’s reliability, security and management of data. And best of all Company D don’t need to change the way they handle data as they can still focus on keeping the file contents updated, but Company A can provide a program that can also embed the MQ MFT Agent which can run and extract the contents of the file and send it as MQ Messages to Company A. Just as with the MQ Client, the MQ MFT Agent is designed for easy embedding in an application, and benefits from also being redistributable under the license. The key difference is that MQ MFT Agents are free but only when they connect to MQ Queue Managers that benefit from the MQ Advanced license entitlement or are in the MQ Appliance. In providing this application making use of the MFT Agent to Company D, Company A is taking advantage of the recent change to make the Agent license redistributable, as well as the fact there is now no cost to embed MFT Agents and distribute them anywhere, as long as they connect to their MQ Advanced Queue Managers. Also, the packaging changed to ensure the MFT Agent was available as a standalone zip file for easier embedding.

 

As a business, your buried treasure may be hidden in your data. You owe it to yourself to ensure it is used as widely as possible and as timely as possible. But to do this you need buried treasure in your applications as well – and this time the buried treasure is the MQ Clients and MQ MFT Agents you can now embed in those applications. Hidden in your code, but providing value every day – maybe not buried treasure, but the goose that lays golden eggs?

Goose Golden Egg

Building higher – IBM MQ V9.0.2

March 16, 2017

When a building is being constructed, it can be hard, from moment to moment to see progress. Yes – you see lots of activity. Lots of people are busy doing all sorts of important jobs, but it can be hard to see what they are all doing. You need to find a way to keep track of how they are doing. What progress are they making, and what milestones are they hitting.

building construction

In delivering updates to IBM MQ, now that we are on a ‘Continuous Delivery’ schedule, we set these milestones of deliveries around 3 times a year. We don’t plan to do IBM announcement letters with every update, but will do blogs here and elsewhere for some of the updates, with official announcements for others. For IBM MQ V9.0.1, there was an announcement letter, and I blogged about it here, but with IBM MQ V9.0.2 there are only blogs – both this one and our development blog from Ian Harwood you can find on developerWorks here. Also there is a YouTube video talking about the new update.

So, what has the development team has been working on in MQ V9.0.2? As with the 9.0.1 update there are several areas of enhancement and new function including:

  • Additional REST API coverage
  • Further updates to the MQ Console
  • Improvements in MQ MFT specifically in MFT Agent status reporting
  • Simplification in managed MQ logging on distributed platforms
  • MQ Appliance support for HA key renewal and 9.0.2 REST API verbs
  • Support for IBM Cloud Product Insights for registration and usage
  • Integration with Salesforce messaging events
  • Native Debian installer support for Ubuntu
  • Availability of MQ Advanced for Developers in the IBM Bluemix Container Service

 

Perhaps as with our description about building construction above, the delivery of any of these features might not be significant, although I think that the logging improvements will make a substantial difference to the many aspects of the use of MQ in the thousands of customers using it today.

 

What hopefully does become apparent is our ongoing support for the continuous delivery process. While some of these updates are brand new and have taken a lot of work, others are continuing to build on the work done in the MQ V9 and MQ V9.0.1 deliveries. These incremental deliveries of REST API support, and now the new Cloud Product Insight support will continue in future Continuous Delivery releases, making these features and the product more useful.

 

Let’s look at a handful of these new features starting with the logging support. Logging is very much the heart of IBM MQ and it is these recovery logs which allow MQ to recover from a failure, therefore providing the reliable and robust nature of IBM MQ. While circular logs are easier to manage, many customer use linear logs but these come with a lot of administrative overhead. The new feature allows for automatic management, recording and reuse of logs, lowering both the administrative overheads and improving the overall throughput in the system

 

IBM Cloud Product Insights is a new cloud hosted offering that many different IBM products will be able to work with. Additional features will be added to work with this over time, but initially there is support for registration and usage. You will be able to register your instances of IBM MQ and track them on the Product Insights dashboard. At this time you will be able to see what level of IBM MQ is install, where, and when it was last running. You will also see some usage information such as the number of persistent and non-persistent messages put, and the total size of data being moved through MQ. There is also a beta of log management, where MQ error logs will be shared with the Product Insights dashboard.

MQSalesforce

You may have seen the recent announcement of IBM and Salesforce working together more closely. We are very pleased that one of the ways this relationship is being demonstrated is through a bridge between Salesforce and MQ. When an event happens in Salesforce such as a change to data or a new application being run (Salesforce Platform Events or PushTopics), there is now the ability to trigger a MQ message to provide information about that event without the MQ application needing to be directly connected to Salesforce, simplifying your environment but making your systems more connected.

 

And finally, we now have a version of MQ Advanced for Developers available in the Bluemix Container Service. This means that the fastest way to create a development environment for IBM MQ might be with a couple of clicks to provision MQ Advanced for Developers. With pre-configured defaults to simplify administration, there has never been an easier way to get started with IBM MQ. What are you waiting for?

Simple can be better – the new MQ and MQ Advanced licensing

January 24, 2017

simplicity

Last year my son did a school project on flight – and his project focused on Leonardo da Vinci, and it was fascinating for us all to learn more about Leonardo’s genius. Not just an artist, his incredible imagination seemed to create and explore new worlds, never dreamed of before. And yet for all his visionary ideas, his quote above also stands out: “Simplicity is the ultimate sophistication”.

The same idea can be seen in Blaise Pascal (and Mark Twain) saying “I didn’t have time to write a short letter, so I wrote a long one instead”. Sadly this applies to this blog entry as well so in the interests of brevity, a quick summary of what’s described in more detail below:
IBM is simplifying the MQ licensing for new purchases:
• Parts now as follows: MQ, MQ Advanced, MQ Idle Standby, MQ Advanced Idle Standby, MQ Advanced for Developers
• MFT Agents are no longer separately and individually licensed but are free to deploy and use when connected to MQ Advanced entitled Queue Managers – essentially providing a free to use MQ MFT network when you use MQ Advanced
• The parts being withdrawn are only those for new entitlements to the separate MQ MFT, MQ AMS and MQ Telemetry parts but not the Subscription and Support renewal parts – you can continue with your existing entitlement as before.
• If you have MQ Advanced today this change applies to all your existing MQ Advanced entitlement – not just to the latest MQ V9.0.1 release.

Today, our world is moving faster and faster. Businesses need to be more agile. Do more with less. Get more for their money. Keeping things simple makes sense today. Even more so as business environments are highly dynamic, and need to balance between unique requirements and common deployments for ease of development, deployment, operations and maintenance.

When it comes to critical offerings like IBM MQ – providing reliable, secure, scalable and robust enterprise messaging, why should we make it more complex than it needs to be? From January 24th 2017, IBM is simplifying the IBM MQ licensing structure to make it simple to describe, simple to purchase, simple to understand and simple to deploy and use.

What are we talking about? Well, for nearly 25 years IBM has been selling IBM MQ – and we still are. But for almost 15 years IBM has been selling extensions to IBM MQ as separate offerings: MQ Managed File Transfer, MQ Advanced Message Security and MQ Telemetry. These all built on and extended the value offered by IBM MQ – and in 2012, as part of MQ V7.5 we brought all the separate components together into a single package, and also created a single offering called MQ Advanced to provide entitlement to the MQ Server along with all of the MQ Server extensions.

Since then, MQ Advanced has been the most popular way to extend MQ, over buying the individual product parts. However, there was always a complexity about the MQ Advanced license for customers using it for Managed File Transfer. This was because MQ’s Managed File Transfer was available as both the MFT Service component that came with MQ Advanced, but also was licensed as MQ MFT Agents on a per Install basis. Even though you might have bought lots of MQ Advanced licenses, you would still need to buy MQ MFT Agents for those systems where you wanted to deploy MQ managed file transfer capabilities, but where you didn’t have MQ Advanced installed. This would be even more noticeable since MQ V9.0.1 shipped which allowed the MQ MFT Agents to be redistributable and made them available in a zip format, suitable for embedding in other solutions. Having per install licensing for MFT Agents would restrict the potential for use of this style of deployment.

MQMFT image

As part of this license change, the MQ MFT Agents are no longer chargeable, or licensed per Install. Instead they are free to deploy and use – in any quantity, as long as the appropriate MQ Servers are licensed with MQ Advanced entitlements. The Agent QM, and the co-ordination QM, and the Logging QM for the MFT Agents must all have MQ Advanced entitlement. These can be all the same Queue Manager, or they can be separated – but all must have MQ Advanced entitlement – but then all MQ MFT Agents using these QMs can be deployed and used at no cost, whether 1 Agent, 100 Agents or more.

mq-new-licenses

The licensing for MQ and MQ Advanced going forward is now very simple. You select IBM MQ if you just want MQ, or IBM MQ Advanced if you want MQ and any other capability. Both are licensed by PVU (perpetual or monthly license) – so by the capacity of the machine where you install the MQ server or by the Virtual Processor Core as described here. Along with IBM MQ and IBM MQ Advanced, there are Idle Standby parts for both, and also IBM MQ Advanced for Developers. Just a handful of parts giving you so much potential for your business.

The additional features in MQ Advanced include MQ Managed File Transfer (as mentioned above) which allows the contents of files to be sent reliably and securely over the MQ network as MQ messages. Differentiating factor with this solution is that the file contents can be directly consumed as messages, moving file transfer into virtually real time data usage. Now available to be deployed anywhere at no additional cost when connected to MQ Advanced Queue Managers. Then there is Advanced Message Security, which provides end to end message content encryption. Since MQ V9 this has a new option allowing for encryption at virtually no impact to performance or throughput, helping you protect your business and customer data from exposure in the case of a breach. And MQ Telemetry which enables your MQ applications to connect directly using the MQTT protocol to mobile phones and the Internet of Things.

It’s all so much simpler now to explain, to buy and to use. But what if you have previously bought some of the separate parts. We have made sure to keep the existing renewal parts available so you can continue to use them and stay current with support on them. So nothing needs to change – you can continue exactly as before. But you might want to consider moving to MQ Advanced entitlement as only this will provide the ability to connect MQ MFT Agents at no cost, and there is no entitlement to buy or deploy new MQ MFT Agents without MQ Advanced entitlement in the future. Existing purchased MQ MFT Agent entitlements remain valid and can continue to be deployed and used.

Feel free to reach out to your IBM rep, your IBM business partner or even me to discuss this, and what it might mean to you. We have tried to do this very carefully so that there is no negative impact on anyone today, and that going forward there are lots of benefits – such as the ability to deploy a much larger MQ managed file transfer network at no additional cost with MQ Advanced entitlement. And as an added change, we have ensure that the MQ Appliance license also allows for connection of MQ MFT Agents at no cost – so that provides an additional deployment and connectivity option for MQ MFT solutions.

Manwithfiles

I will try to write another blog shortly about our MQ Managed File Transfer solution soon – but this one needs to end so you can get back to work.

Think what you can do with this now. It’s going to be a busy year. Let’s start now.